{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25141ed7",
   "metadata": {},
   "source": [
    "# 预训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8c582b6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matplotlib version: 3.10.3\n",
      "numpy version: 2.0.2\n",
      "tiktoken version: 0.9.0\n",
      "torch version: 2.7.1\n",
      "tensorflow version: 2.19.0\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"matplotlib\", \n",
    "        \"numpy\", \n",
    "        \"tiktoken\", \n",
    "        \"torch\",\n",
    "        \"tensorflow\" # For OpenAI's pretrained weights\n",
    "       ]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33b405b1",
   "metadata": {},
   "source": [
    "![](./images/overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7239169f",
   "metadata": {},
   "source": [
    "- 在本章中，我们实现了用于预训练大语言模型（LLM）的训练循环及基础模型评估代码；\n",
    "- 在本章末尾，我们还将OpenAI公开提供的预训练权重加载到我们的模型中；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e43da8",
   "metadata": {},
   "source": [
    "![](./images/scenario.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa3c125",
   "metadata": {},
   "source": [
    "## 生成式文本模型评估"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56dbdfaa",
   "metadata": {},
   "source": [
    "- 本节首先简要回顾如何使用前一章的代码初始化GPT模型；\n",
    "- ​​接着探讨用于大语言模型（LLMs）的基本评估指标，包括困惑度（Perplexity）和交叉熵损失（Cross-Entropy Loss）等量化工具；\n",
    "- ​最后，我们将这些评估指标应用于训练集和验证数据集，以量化模型生成文本的质量；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7d3944ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "from utils import GPTModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b303c316",
   "metadata": {},
   "outputs": [],
   "source": [
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,   # Vocabulary size\n",
    "    \"context_length\": 128, # Shortened context length (orig: 1024)\n",
    "    \"emb_dim\": 768,        # Embedding dimension\n",
    "    \"n_heads\": 12,         # Number of attention heads\n",
    "    \"n_layers\": 12,        # Number of layers\n",
    "    \"drop_rate\": 0.1,      # Dropout rate\n",
    "    \"qkv_bias\": False      # Query-key-value bias\n",
    "}\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.eval();  # Disable dropout during inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7a7ec76",
   "metadata": {},
   "source": [
    "- 上文我们设置了0.1的丢弃率，但需指出的是，​​当前训练大语言模型时完全弃用丢弃法已相对常见​​；\n",
    "- 现代大语言模型在查询、键、值矩阵对应的nn.Linear层中​​通常不再使用偏置向量​​（与早期GPT模型不同），这可通过设置\"qkv_bias\": False实现；\n",
    "- 为降低模型训练的计算资源需求，我们将上下文长度限制为128个词元，而原始1.24亿参数GPT-2模型使用1024个词元；\n",
    "    - 此设置旨在能够在个人笔记本电脑上顺利运行代码示例；\n",
    "    - 可将context_length增至1024词元（​​无需修改代码逻辑​​）；\n",
    "    - 我们后续也将加载上下文长度为1024的预训练权重模型；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35d14834",
   "metadata": {},
   "source": [
    "### 使用GPT产生文本"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98e8d31d",
   "metadata": {},
   "source": [
    "- 接下来，我们使用前一章实现的generate_text_simple函数进行文本生成；\n",
    "- 此外，我们定义了两个便利函数text_to_token_ids和token_ids_to_text，用于实现文本与标记表示之间的双向转换，本章将全程使用这些转换工具；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e44bca0",
   "metadata": {},
   "source": [
    "![](./images/gpt-process.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f08f722f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a68f3632",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_simple(model, idx, max_new_tokens, context_size):\n",
    "    # idx is (B, T) array of indices in the current context\n",
    "    for _ in range(max_new_tokens):\n",
    "\n",
    "        # Crop current context if it exceeds the supported context size\n",
    "        # E.g., if LLM supports only 5 tokens, and the context size is 10\n",
    "        # then only the last 5 tokens are used as context\n",
    "        idx_cond = idx[:, -context_size:]\n",
    "\n",
    "        # Get the predictions\n",
    "        with torch.no_grad():\n",
    "            logits = model(idx_cond)\n",
    "\n",
    "        # Focus only on the last time step\n",
    "        # (batch, n_token, vocab_size) becomes (batch, vocab_size)\n",
    "        logits = logits[:, -1, :]\n",
    "\n",
    "        # Get the idx of the vocab entry with the highest logits value\n",
    "        idx_next = torch.argmax(logits, dim=-1, keepdim=True)  # (batch, 1)\n",
    "\n",
    "        # Append sampled index to the running sequence\n",
    "        idx = torch.cat((idx, idx_next), dim=1)  # (batch, n_tokens+1)\n",
    "\n",
    "    return idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ddf51019",
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_to_token_ids(text, tokenizer):\n",
    "    encoded = tokenizer.encode(text, allowed_special={\"<|endoftext|>\"})\n",
    "    encoded_tensor = torch.tensor(encoded).unsqueeze(0)\n",
    "\n",
    "    return encoded_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "98184780",
   "metadata": {},
   "outputs": [],
   "source": [
    "def token_ids_to_text(token_ids, tokenizer):\n",
    "    flat = token_ids.squeeze(0)\n",
    "    return tokenizer.decode(flat.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fde7a09b",
   "metadata": {},
   "outputs": [],
   "source": [
    "start_context = \"Every effort moves you\"\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(start_context, tokenizer),\n",
    "    max_new_tokens=10,\n",
    "    context_size=GPT_CONFIG_124M[\"context_length\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "281d03b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Every effort moves you Spirits caves Feng lith cohorts Physical Position Wind Bars Vern'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids_to_text(token_ids, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88873b4",
   "metadata": {},
   "source": [
    "### 文本生成损失计算：交叉熵与困惑度"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caee62fa",
   "metadata": {},
   "source": [
    "- 假设我们有一个包含2个训练样本（行）标记ID的inputs张量；\n",
    "- ​与inputs相对应，targets包含了我们希望模型生成的期望标记ID；\n",
    "- 值得注意的是，targets是inputs向右偏移1个位置的序列，正如我们在第2章实现数据加载器时所解释的；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "af66610d",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = torch.tensor([[16833, 3626, 6100],   # [\"every effort moves\",\n",
    "                       [40,    1107, 588]])   #  \"I really like\"]\n",
    "\n",
    "targets = torch.tensor([[3626, 6100, 345  ],  # [\" effort moves you\",\n",
    "                        [1107,  588, 11311]]) #  \" really like chocolate\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eeed0c2",
   "metadata": {},
   "source": [
    "- 将inputs输入模型后，我们获得了对应2个输入样本的logits向量，每个样本由3个词元组成；\n",
    "- 每个词元对应一个50,257维的向量，该维度与词汇表的大小一致；\n",
    "- 通过应用softmax函数，我们可以将logits张量转换为相同维度的概率得分张量；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "482526be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 50257])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "with torch.no_grad():\n",
    "    logits = model(inputs)\n",
    "\n",
    "logits.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e264e40f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.2135, -0.3297,  0.1896,  ...,  0.5931, -0.0786,  0.4132])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7411b988",
   "metadata": {},
   "outputs": [],
   "source": [
    "probs = torch.softmax(logits, dim=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3d5fbe06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 50257])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b40812b4",
   "metadata": {},
   "source": [
    "- 下图通过一个极简词汇表示例，直观展示了如何将模型输出的概率得分还原为文本序列，此项转换流程我们在前一章末尾已进行过详细探讨；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05aefacd",
   "metadata": {},
   "source": [
    "![](./images/proba-to-text.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "683da101",
   "metadata": {},
   "source": [
    "inputs = torch.tensor([[16833, 3626, 6100],   # [\"every effort moves\",\n",
    "                       [40,    1107, 588]])   #  \"I really like\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "928d61eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[38862],\n",
       "         [43027],\n",
       "         [ 8569]],\n",
       "\n",
       "        [[42085],\n",
       "         [31970],\n",
       "         [16774]]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids = torch.argmax(probs, dim=-1, keepdim=True)\n",
    "token_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "04120a1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' effort moves you'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids_to_text(targets[0], tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "87d534bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([38862, 43027,  8569])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids[0].flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8aea2e25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' doverank comprehens'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids_to_text(token_ids[0].flatten(), tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f755a211",
   "metadata": {},
   "source": [
    "![](./images/proba-index.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8c3325cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3.3706e-05, 2.9752e-05, 1.4406e-05])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_id = 0\n",
    "target_probs_1 = probs[text_id, [0, 1, 2], targets[text_id]]\n",
    "\n",
    "target_probs_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1477ed71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.1550e-05, 1.7313e-05, 1.7490e-05])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_id = 1\n",
    "target_probs_2 = probs[text_id, [0, 1, 2], targets[text_id]]\n",
    "\n",
    "target_probs_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5fa8e534",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-10.2978, -10.4226, -11.1479, -10.7451, -10.9641, -10.9539])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_probs = torch.log(torch.cat((target_probs_1, target_probs_2)))\n",
    "log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2b497412",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(10.7552)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avg_log_probs = torch.mean(log_probs)\n",
    "\n",
    "avg_log_probs * (-1.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1be691ba",
   "metadata": {},
   "source": [
    "![](./images/cross-entropy.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "033a6682",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 50257])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a9bf7313",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 50257])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits_flat = logits.flatten(0, 1)\n",
    "logits_flat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c937a5bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targets.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f81d68b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targets_flat = targets.flatten(0, 1)\n",
    "targets_flat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "43997ef8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(10.7552)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = torch.nn.functional.cross_entropy(logits_flat, targets_flat)\n",
    "\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da292bc0",
   "metadata": {},
   "source": [
    "困惑度（Perplexity）是评估语言模型性能的核心指标之一，它衡量了模型在预测序列数据时的不确定性程度。简单来说，​​困惑度越低，说明模型对数据的预测越准确、越自信​​；困惑度越高，则代表模型越“困惑”，预测能力越差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "66c6401a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(46875.0781)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perplexity = torch.exp(loss)\n",
    "perplexity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "be322186",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 50257])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a47e44a",
   "metadata": {},
   "source": [
    "### 计算训练损失和验证损失"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c96f79",
   "metadata": {},
   "source": [
    "- 我们采用一个​​相对小规模的数据集​​用于大语言模型训练（实际上仅包含一篇短篇故事）；\n",
    "- 此设计基于以下几点考量：\n",
    "    - 确保模型可在​​笔记本电脑​​上（无需高性能GPU）​​数分钟内完成代码运行​​；\n",
    "    - 实现​​快速训练​​（仅需数分钟而非数周），契合教学演示需求；\n",
    "    - 选用公共领域文本​​，避免版权争议且不会过度增加代码仓库体积；\n",
    "\n",
    "- 以Llama 2 7B模型为例：其训练消耗​​184320个A100 GPU小时​​，数据处理量达​​2万亿词元​​；\n",
    "    - 当前8xA100云服务器每小时成本约​​30美元​​；\n",
    "    - 经粗略估算：训练总成本约为 184320 ÷ 8 × 30 = ​​69万美元​​\n",
    "\n",
    "- 下文将延续使用第2章中的同一数据集；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "01756df5",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = \"corpus.txt\"\n",
    "with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "    text_data = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4d762532",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: 20479\n",
      "Tokens: 5145\n"
     ]
    }
   ],
   "source": [
    "total_characters = len(text_data)\n",
    "total_tokens = len(tokenizer.encode(text_data))\n",
    "\n",
    "print(\"Characters:\", total_characters)\n",
    "print(\"Tokens:\", total_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "3aefe964",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[40,\n",
       " 367,\n",
       " 2885,\n",
       " 1464,\n",
       " 1807,\n",
       " 3619,\n",
       " 402,\n",
       " 271,\n",
       " 10899,\n",
       " 2138,\n",
       " 257,\n",
       " 7026,\n",
       " 15632,\n",
       " 438,\n",
       " 2016,\n",
       " 257,\n",
       " 922,\n",
       " 5891,\n",
       " 1576,\n",
       " 438,\n",
       " 568,\n",
       " 340,\n",
       " 373,\n",
       " 645,\n",
       " 1049,\n",
       " 5975,\n",
       " 284,\n",
       " 502,\n",
       " 284,\n",
       " 3285,\n",
       " 326,\n",
       " 11,\n",
       " 287,\n",
       " 262,\n",
       " 6001,\n",
       " 286,\n",
       " 465,\n",
       " 13476,\n",
       " 11,\n",
       " 339,\n",
       " 550,\n",
       " 5710,\n",
       " 465,\n",
       " 12036,\n",
       " 11,\n",
       " 6405,\n",
       " 257,\n",
       " 5527,\n",
       " 27075,\n",
       " 11]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.encode(text_data)[:50]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5301cd88",
   "metadata": {},
   "source": [
    "- 该文本仅含5145个词元，对于大语言模型训练而言长度极短，仅用于教学目的（我们后续将加载预训练权重）；\n",
    "- 接下来，我们将数据集划分为训练集和验证集，并沿用第2章的数据加载器来准备LLM训练所需的批次数据；\n",
    "- 为便于可视化演示，下图假设设置max_length=6，但实际训练加载器中，我们将max_length设置为LLM所支持的上下文长度；\n",
    "- 为简化示意图，下图仅展示输入词元\n",
    "    - 由于我们训练LLM的目标是预测文本中的下一个词，因此目标张量与这些输入的形状相同，仅存在向右偏移一个位置的差异\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba081bdd",
   "metadata": {},
   "source": [
    "![](./images/batching.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "647768a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from codes import create_dataloader_v1\n",
    "\n",
    "train_ratio = 0.90\n",
    "split_idx = int(train_ratio * len(text_data))\n",
    "train_data = text_data[:split_idx]\n",
    "val_data = text_data[split_idx:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1aaf6ba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = create_dataloader_v1(\n",
    "    train_data,\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=True,\n",
    "    shuffle=True,\n",
    "    num_workers=0\n",
    ")\n",
    "\n",
    "val_loader = create_dataloader_v1(\n",
    "    val_data,\n",
    "    batch_size=2,\n",
    "    max_length=GPT_CONFIG_124M[\"context_length\"],\n",
    "    stride=GPT_CONFIG_124M[\"context_length\"],\n",
    "    drop_last=False,\n",
    "    shuffle=False,\n",
    "    num_workers=0\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "d60b53a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loader:\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "\n",
      "Validation loader:\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n",
      "torch.Size([2, 128]) torch.Size([2, 128])\n"
     ]
    }
   ],
   "source": [
    "print(\"Train loader:\")\n",
    "for x, y in train_loader:\n",
    "    print(x.shape, y.shape)\n",
    "\n",
    "print(\"\\nValidation loader:\")\n",
    "for x, y in val_loader:\n",
    "    print(x.shape, y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "fd7bf4b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training tokens: 4608\n",
      "Validation tokens: 512\n",
      "All tokens: 5120\n"
     ]
    }
   ],
   "source": [
    "train_tokens = 0\n",
    "for input_batch, target_batch in train_loader:\n",
    "    train_tokens += input_batch.numel()\n",
    "\n",
    "val_tokens = 0\n",
    "for input_batch, target_batch in val_loader:\n",
    "    val_tokens += input_batch.numel()\n",
    "\n",
    "print(\"Training tokens:\", train_tokens)\n",
    "print(\"Validation tokens:\", val_tokens)\n",
    "print(\"All tokens:\", train_tokens + val_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e3fac5cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_loss_batch(input_batch, target_batch, model, device):\n",
    "    input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "    logits = model(input_batch)\n",
    "    loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n",
    "    return loss\n",
    "\n",
    "\n",
    "def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
    "    total_loss = 0.\n",
    "    if len(data_loader) == 0:\n",
    "        return float(\"nan\")\n",
    "    elif num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        # Reduce the number of batches to match the total number of batches in the data loader\n",
    "        # if num_batches exceeds the number of batches in the data loader\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            total_loss += loss.item()\n",
    "        else:\n",
    "            break\n",
    "    return total_loss / num_batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "375db1e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 11.012801806131998\n",
      "Validation loss: 10.996222019195557\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model.to(device)\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n",
    "    train_loss = calc_loss_loader(train_loader, model, device)\n",
    "    val_loss = calc_loss_loader(val_loader, model, device)\n",
    "\n",
    "print(\"Training loss:\", train_loss)\n",
    "print(\"Validation loss:\", val_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "7b79c3ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.8221)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perplexity = torch.exp(torch.tensor(0.6))\n",
    "\n",
    "perplexity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43d962b2",
   "metadata": {},
   "source": [
    "![](./images/mental-model-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "386d07fc",
   "metadata": {},
   "source": [
    "## 训练大语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9450f30f",
   "metadata": {},
   "source": [
    "- 在本节中，我们最终实现用于训练大语言模型（LLM）的代码；\n",
    "- ​我们重点实现一个基础的训练函数；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173b50ef",
   "metadata": {},
   "source": [
    "![](./images/train-steps.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "6e26ab5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
    "                       eval_freq, eval_iter, start_context, tokenizer):\n",
    "    train_losses, val_losses, track_tokens_seen = [], [], []\n",
    "    tokens_seen, global_step = 0, -1\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()\n",
    "        \n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            tokens_seen += input_batch.numel()\n",
    "            global_step += 1\n",
    "\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader, device, eval_iter)\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                track_tokens_seen.append(tokens_seen)\n",
    "                print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        generate_and_print_sample(\n",
    "            model, tokenizer, device, start_context\n",
    "        )\n",
    "\n",
    "    return train_losses, val_losses, track_tokens_seen\n",
    "\n",
    "\n",
    "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "    model.train()\n",
    "    return train_loss, val_loss\n",
    "\n",
    "\n",
    "def generate_and_print_sample(model, tokenizer, device, start_context):\n",
    "    model.eval()\n",
    "    context_size = model.pos_emb.weight.shape[0]\n",
    "    encoded = text_to_token_ids(start_context, tokenizer).to(device)\n",
    "    with torch.no_grad():\n",
    "        token_ids = generate_text_simple(\n",
    "            model=model, idx=encoded,\n",
    "            max_new_tokens=50, context_size=context_size\n",
    "        )\n",
    "    decoded_text = token_ids_to_text(token_ids, tokenizer)\n",
    "    print(decoded_text.replace(\"\\n\", \" \"))\n",
    "    model.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ed40ddbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Step 000000): Train loss 9.997, Val loss 10.184\n",
      "Ep 1 (Step 000005): Train loss 8.379, Val loss 8.522\n",
      "Ep 1 (Step 000010): Train loss 7.169, Val loss 7.415\n",
      "Ep 1 (Step 000015): Train loss 6.190, Val loss 6.814\n",
      "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n",
      "Ep 2 (Step 000020): Train loss 6.188, Val loss 6.593\n",
      "Ep 2 (Step 000025): Train loss 5.706, Val loss 6.538\n",
      "Ep 2 (Step 000030): Train loss 6.003, Val loss 6.541\n",
      "Ep 2 (Step 000035): Train loss 5.826, Val loss 6.538\n",
      "Every effort moves you the the                                                \n",
      "Ep 3 (Step 000040): Train loss 5.562, Val loss 6.524\n",
      "Ep 3 (Step 000045): Train loss 5.656, Val loss 6.518\n",
      "Ep 3 (Step 000050): Train loss 5.481, Val loss 6.438\n",
      "Every effort moves you.                                                 \n",
      "Ep 4 (Step 000055): Train loss 5.061, Val loss 6.438\n",
      "Ep 4 (Step 000060): Train loss 5.018, Val loss 6.309\n",
      "Ep 4 (Step 000065): Train loss 4.669, Val loss 6.328\n",
      "Ep 4 (Step 000070): Train loss 4.607, Val loss 6.244\n",
      "Every effort moves you, and in the picture of the picture--I the picture. Gisburn. Gisburn's--I in the picture--I had been of his his painting, and I was his painting, and his a little, and in the picture\n",
      "Ep 5 (Step 000075): Train loss 4.150, Val loss 6.199\n",
      "Ep 5 (Step 000080): Train loss 3.418, Val loss 6.135\n",
      "Ep 5 (Step 000085): Train loss 3.803, Val loss 6.128\n",
      "Every effort moves you of the fact, I felt to have to the picture--as, with a little, with a little, he had been, with a little, he had the fact of the picture, with a little of the picture--because he had the picture\n",
      "Ep 6 (Step 000090): Train loss 2.903, Val loss 6.132\n",
      "Ep 6 (Step 000095): Train loss 2.856, Val loss 6.090\n",
      "Ep 6 (Step 000100): Train loss 2.775, Val loss 6.124\n",
      "Ep 6 (Step 000105): Train loss 2.432, Val loss 6.162\n",
      "Every effort moves you know,\" he was a little a little quickly.                              \"Oh, and he was a a a little\n",
      "Ep 7 (Step 000110): Train loss 2.172, Val loss 6.077\n",
      "Ep 7 (Step 000115): Train loss 1.808, Val loss 6.164\n",
      "Ep 7 (Step 000120): Train loss 1.611, Val loss 6.245\n",
      "Ep 7 (Step 000125): Train loss 1.296, Val loss 6.199\n",
      "Every effort moves you know,\" was not that the house.\"  \"I was such me--and that with a dozen lines--I didn't you know, and threw back the same quality as his pictures--the quality of Jack's \"strongest,\" she said\n",
      "Ep 8 (Step 000130): Train loss 1.221, Val loss 6.209\n",
      "Ep 8 (Step 000135): Train loss 1.159, Val loss 6.226\n",
      "Ep 8 (Step 000140): Train loss 0.876, Val loss 6.338\n",
      "Every effort moves you?\"  \"I turned to put it--_I had never known_.. I said. I looked up the fact brought home to me the absolute finality of Jack's break with his old life.    I was not\n",
      "Ep 9 (Step 000145): Train loss 0.742, Val loss 6.387\n",
      "Ep 9 (Step 000150): Train loss 0.636, Val loss 6.449\n",
      "Ep 9 (Step 000155): Train loss 0.461, Val loss 6.505\n",
      "Ep 9 (Step 000160): Train loss 0.546, Val loss 6.440\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. Rickham--the loss to Arrt is all I think of.\"  \"Oh, I looked at the donkey again. I saw that, and down the donkey hanging on the\n",
      "Ep 10 (Step 000165): Train loss 0.410, Val loss 6.541\n",
      "Ep 10 (Step 000170): Train loss 0.390, Val loss 6.643\n",
      "Ep 10 (Step 000175): Train loss 0.328, Val loss 6.692\n",
      "Every effort moves you?\"  \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"    I moved away, one could always get near enough to see his pictures. \"The last but one,\" she corrected\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.to(device)\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1)\n",
    "\n",
    "num_epochs = 10\n",
    "train_losses, val_losses, tokens_seen = train_model_simple(\n",
    "    model, train_loader, val_loader, optimizer, device,\n",
    "    num_epochs=num_epochs, eval_freq=5, eval_iter=5,\n",
    "    start_context=\"Every effort moves you\", tokenizer=tokenizer\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26ab0bfc",
   "metadata": {},
   "source": [
    "- 计算机上可能会得到略有不同的损失值，若其大致相似则无需担忧；\n",
    "- 微小差异通常源于不同的GPU硬件与CUDA版本，或新版PyTorch中的细微改动；\n",
    "- 即使在CPU上运行示例，也可能观察到差异；一个可能的原因是nn.Dropout在不同操作系统间的行为差异，这取决于PyTorch的编译方式；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "cff52acf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAWX1JREFUeJzt3XlYE1fbB+BfEkhIgIR9k1VFNhFQwCLaakVxqXWty8vb4lJtK261WmutW1trrdZarZ/W9q12catW3De0ioobLiAWRFQ2lUVEdgiQnO+PkWAKKiCQgM99XXMlOXNm5skQ8uScOTPDY4wxEEIIIUQr8TUdACGEEEKejhI1IYQQosUoURNCCCFajBI1IYQQosUoURNCCCFajBI1IYQQosUoURNCCCFajBI1IYQQosUoURNCCCFajBI1Ia1ASkoKeDweYmJiNB0KIaSRUaImREvweLxnTosWLdJ0iIQQDdDRdACEEE5GRobq+fbt27FgwQIkJiaqygwMDDQRFiFEw6hFTYiWsLKyUk0ymQw8Hk/12sLCAitXroStrS1EIhG8vb1x+PDhp65LoVBg/PjxcHV1RVpaGgBgz5496Ny5M/T09NC2bVssXrwYlZWVqmV4PB5+/vlnDB06FBKJBM7Ozti7d69q/qNHjxASEgJzc3OIxWI4Oztj48aNT41h586d8PT0hFgshqmpKYKCglBcXKya//PPP8PNzQ16enpwdXXF//3f/6ktn56ejpEjR8LIyAgmJiYYPHgwUlJSVPPHjh2LIUOGYMWKFbC2toapqSnCwsJQUVFR531OSIvACCFaZ+PGjUwmk6ler1y5kkmlUrZ161Z248YN9vHHHzNdXV128+ZNxhhjycnJDAC7evUqKysrY0OHDmU+Pj4sOzubMcbYqVOnmFQqZZs2bWK3b99mR48eZY6OjmzRokWqbQBgtra2bMuWLSwpKYlNmzaNGRgYsIcPHzLGGAsLC2Pe3t4sOjqaJScns4iICLZ3795a479//z7T0dFhK1euZMnJyezatWts7dq1rLCwkDHG2B9//MGsra3ZX3/9xe7cucP++usvZmJiwjZt2sQYY6y8vJy5ubmx8ePHs2vXrrH4+Hj2n//8h7m4uDC5XM4YYyw0NJRJpVL2/vvvs4SEBLZv3z4mkUjYhg0bGvePQYiGUaImRAv9O1Hb2NiwJUuWqNXx8/NjkydPZoxVJ+rTp0+z3r17s+7du7O8vDxV3d69e7OvvvpKbfnff/+dWVtbq14DYJ999pnqdVFREQPADh06xBhjbNCgQWzcuHF1iv/y5csMAEtJSal1frt27diWLVvUyr744gsWEBCgis3FxYUplUrVfLlczsRiMTty5AhjjEvUDg4OrLKyUlXnrbfeYqNGjapTjIS0FHSMmhAtV1BQgPv37yMwMFCtPDAwELGxsWplY8aMga2tLf7++2+IxWJVeWxsLKKiorBkyRJVmUKhQFlZGUpKSiCRSAAAnTp1Us3X19eHVCpFdnY2AOCDDz7A8OHDceXKFfTt2xdDhgxBt27dao3Zy8sLvXv3hqenJ4KDg9G3b1+MGDECxsbGKC4uxu3btzFhwgRMnDhRtUxlZSVkMpkq3lu3bsHQ0FBtvWVlZbh9+7bqtYeHBwQCgeq1tbU14uLinrE3CWl5KFET0ooMGDAAf/zxB86dO4fXX39dVV5UVITFixdj2LBhNZbR09NTPdfV1VWbx+PxoFQqAQD9+/dHamoqDh48iIiICPTu3RthYWFYsWJFjXUKBAJERETg7NmzOHr0KNasWYN58+bhwoULqh8FP/30E7p27Vpjuap4u3Tpgs2bN9dYt7m5eZ3iJaS1oERNiJaTSqWwsbFBVFQUXnvtNVV5VFQU/P391ep+8MEH6NixI958800cOHBAVb9z585ITExE+/btXygWc3NzhIaGIjQ0FD169MDs2bNrTdQAlzQDAwMRGBiIBQsWwMHBAeHh4Zg5cyZsbGxw584dhISE1Lps586dsX37dlhYWEAqlb5QzIS0dJSoCWkBZs+ejYULF6Jdu3bw9vbGxo0bERMTU2uLc+rUqVAoFHjjjTdw6NAhdO/eHQsWLMAbb7wBe3t7jBgxAnw+H7Gxsbh+/Tq+/PLLOsWwYMECdOnSBR4eHpDL5di/fz/c3NxqrXvhwgUcP34cffv2hYWFBS5cuIAHDx6o6i9evBjTpk2DTCZDv379IJfLcenSJTx69AgzZ85ESEgIli9fjsGDB+Pzzz+Hra0tUlNTsWvXLnz88cewtbVt+M4kpIWhRE1ICzBt2jTk5+fjo48+QnZ2Ntzd3bF37144OzvXWn/GjBlQKpUYMGAADh8+jODgYOzfvx+ff/45li1bBl1dXbi6uuLdd9+tcwxCoRBz585FSkoKxGIxevTogW3bttVaVyqV4tSpU1i1ahUKCgrg4OCAb7/9Fv379wcAvPvuu5BIJFi+fDlmz54NfX19eHp6YsaMGQAAiUSCU6dOYc6cORg2bBgKCwvRpk0b9O7dm1rY5KXDY4wxTQdBCCGEkNrRBU8IIYQQLUaJmhBCCNFilKgJIYQQLUaJmhBCCNFilKgJIYQQLUaJmhBCCNFilKifY+3atXB0dISenh66du2KixcvajokjTl16hQGDRoEGxsb8Hg87N69W20+YwwLFiyAtbU1xGIxgoKCkJSUpFYnNzcXISEhkEqlMDIywoQJE1BUVKRW59q1a+jRowf09PRgZ2eHb775pkYsO3bsgKurK/T09ODp6YmDBw82+vttLkuXLoWfnx8MDQ1hYWGBIUOGqN2HGuCucR0WFgZTU1MYGBhg+PDhyMrKUquTlpaGgQMHQiKRwMLCArNnz1a7jSUAnDx5Ep07d4ZIJEL79u2xadOmGvG0ls/8unXr0KlTJ0ilUkilUgQEBODQoUOq+bRPX9zXX38NHo+nOv8doP3aJDR8UxCttm3bNiYUCtkvv/zC/vnnHzZx4kRmZGTEsrKyNB2aRhw8eJDNmzeP7dq1iwFg4eHhavO//vprJpPJ2O7du1lsbCx78803mZOTEystLVXV6devH/Py8mLnz59np0+fZu3bt2djxoxRzc/Pz2eWlpYsJCSEXb9+nW3dupWJxWL2448/qupERUUxgUDAvvnmGxYfH88+++wzpqury+Li4pp8HzSF4OBgtnHjRnb9+nUWExPDBgwYwOzt7VlRUZGqzvvvv8/s7OzY8ePH2aVLl9grr7zCunXrpppfWVnJOnbsyIKCgtjVq1fZwYMHmZmZGZs7d66qzp07d5hEImEzZ85k8fHxbM2aNUwgELDDhw+r6rSmz/zevXvZgQMH2M2bN1liYiL79NNPma6uLrt+/TpjjPbpi7p48SJzdHRknTp1YtOnT1eV035tfJSon8Hf35+FhYWpXisUCmZjY8OWLl2qwai0w78TtVKpZFZWVmz58uWqsry8PCYSidjWrVsZY4zFx8czACw6OlpV59ChQ4zH47F79+4xxhj7v//7P2ZsbKy65zBjjM2ZM4e5uLioXo8cOZINHDhQLZ6uXbuy9957r1Hfo6ZkZ2czACwyMpIxxu1HXV1dtmPHDlWdhIQEBoCdO3eOMcb9iOLz+SwzM1NVZ926dUwqlar25ccff8w8PDzUtjVq1CgWHByset3aP/PGxsbs559/pn36ggoLC5mzszOLiIhgr732mipR035tGtT1/RTl5eW4fPkygoKCVGV8Ph9BQUE4d+6cBiPTTsnJycjMzFTbXzKZDF27dlXtr3PnzsHIyAi+vr6qOkFBQeDz+bhw4YKqzquvvgqhUKiqExwcjMTERDx69EhV58ntVNVpLX+X/Px8AICJiQkA4PLly6ioqFB7z66urrC3t1fbt56enrC0tFTVCQ4ORkFBAf755x9VnWftt9b8mVcoFNi2bRuKi4sREBBA+/QFhYWFYeDAgTXeO+3XpkHX+n6KnJwcKBQKtQ8TAFhaWuLGjRsaikp7ZWZmAkCt+6tqXmZmJiwsLNTm6+jowMTERK2Ok5NTjXVUzTM2NkZmZuYzt9OSKZVKzJgxA4GBgejYsSMA7n0LhUIYGRmp1f33vq1tn1TNe1adgoIClJaW4tGjR63uMx8XF4eAgACUlZXBwMAA4eHhcHd3R0xMDO3TBtq2bRuuXLmC6OjoGvPos9o0KFETokXCwsJw/fp1nDlzRtOhtAouLi6IiYlBfn4+du7cidDQUERGRmo6rBYrPT0d06dPR0REhNp9zEnToq7vpzAzM4NAIKgxWjErKwtWVlYaikp7Ve2TZ+0vKysrZGdnq82vrKxEbm6uWp3a1vHkNp5Wp6X/XaZMmYL9+/fjxIkTardxtLKyQnl5OfLy8tTq/3vfNnS/SaVSiMXiVvmZFwqFaN++Pbp06YKlS5fCy8sL33//Pe3TBrp8+TKys7PRuXNn6OjoQEdHB5GRkVi9ejV0dHRgaWlJ+7UJUKJ+CqFQiC5duuD48eOqMqVSiePHjyMgIECDkWknJycnWFlZqe2vgoICXLhwQbW/AgICkJeXh8uXL6vq/P3331AqlejatauqzqlTp1BRUaGqExERARcXFxgbG6vqPLmdqjot9e/CGMOUKVMQHh6Ov//+u0bXf5cuXaCrq6v2nhMTE5GWlqa2b+Pi4tR+CEVEREAqlcLd3V1V51n77WX4zCuVSsjlctqnDdS7d2/ExcUhJiZGNfn6+iIkJET1nPZrE9D0aDZttm3bNiYSidimTZtYfHw8mzRpEjMyMlIbrfgyKSwsZFevXmVXr15lANjKlSvZ1atXWWpqKmOMOz3LyMiI7dmzh127do0NHjy41tOzfHx82IULF9iZM2eYs7Oz2ulZeXl5zNLSkr399tvs+vXrbNu2bUwikdQ4PUtHR4etWLGCJSQksIULF7bo07M++OADJpPJ2MmTJ1lGRoZqKikpUdV5//33mb29Pfv777/ZpUuXWEBAAAsICFDNrzrlpW/fviwmJoYdPnyYmZub13rKy+zZs1lCQgJbu3Ztrae8tJbP/CeffMIiIyNZcnIyu3btGvvkk08Yj8djR48eZYzRPm0sT476Zoz2a1OgRP0ca9asYfb29kwoFDJ/f392/vx5TYekMSdOnGAAakyhoaGMMe4Urfnz5zNLS0smEolY7969WWJioto6Hj58yMaMGcMMDAyYVCpl48aNY4WFhWp1YmNjWffu3ZlIJGJt2rRhX3/9dY1Y/vzzT9ahQwcmFAqZh4cHO3DgQJO976ZW2z4FwDZu3KiqU1payiZPnsyMjY2ZRCJhQ4cOZRkZGWrrSUlJYf3792disZiZmZmxjz76iFVUVKjVOXHiBPP29mZCoZC1bdtWbRtVWstnfvz48czBwYEJhUJmbm7OevfurUrSjNE+bSz/TtS0XxsfjzHGNNOWJ4QQQsjz0DFqQgghRItRoiaEEEK0GCVqQgghRItRoiaEEEK0GCVqQgghRItRoiaEEEK0GCXqOpDL5Vi0aBHkcrmmQ2lVaL82DdqvTYP2a+OjfVo3dB51HRQUFEAmkyE/Px9SqVTT4bQatF+bBu3XpkH7tfHRPq0balETQgghWowSNSGEEKLFWv39qCsrK3H16lVYWlqCz2/Y75LCwkIAwL1791BQUNCY4b3UaL82DdqvTYP2a+N7mfepUqlEVlYWfHx8oKPz7FTc6o9RR0dHw9/fX9NhEEIIITVcvHgRfn5+z6zT6lvUlpaWALidYW1treFoCCGEECAjIwP+/v6qHPUsrT5RV3V3W1tbw9bWVsPREEIIIdXqckiWBpMRQgghWowSNSGEEKLFKFETQgghWqzVH6MmhJD6UCgUqKio0HQYpIXT1dWFQCBolHVpNFGfOnUKy5cvx+XLl5GRkYHw8HAMGTJENZ8xhoULF+Knn35CXl4eAgMDsW7dOjg7O2suaEJIq8QYQ2ZmJvLy8jQdCmkljIyMYGVlBR6P90Lr0WiiLi4uhpeXF8aPH49hw4bVmP/NN99g9erV+PXXX+Hk5IT58+cjODgY8fHx0NPTa/6AywqAf8IBx+6Aabvm3z4hpMlUJWkLCwtIJJIX/nIlLy/GGEpKSpCdnQ0AL3xqsEYTdf/+/dG/f/9a5zHGsGrVKnz22WcYPHgwAOC3336DpaUldu/ejdGjRzdnqJw9YUDCXiBwOtDn8+bfPiGkSSgUClWSNjU11XQ4pBUQi8UAgOzsbFhYWLxQN7jWDiZLTk5GZmYmgoKCVGUymQxdu3bFuXPnNBKTwnMU9yRmK6CgY1iEtBZVx6QlEomGIyGtSdXn6UXHPGhtos7MzASAGldtsbS0VM2rjVwuR0FBgWqqupbsi3pUXI4e4TrIYVKgOBtIimiU9RJCtAd1d5PG1FifJ61N1A21dOlSyGQy1eTu7t4o6zXWF8LCyBB/KXpwBVf/aJT1EkIIIc+itYnaysoKAJCVlaVWnpWVpZpXm7lz5yI/P181xcfHN1pM//G3x5+KngAAdvMwUJj17AUIIaQFcnR0xKpVq+pc/+TJk+DxeE0+Yn7Tpk0wMjJq0m1oI61N1E5OTrCyssLx48dVZQUFBbhw4QICAgKeupxIJIJUKlVNhoaGjRbTG17WyBI64LLSGTymAK5ta7R1E0JIffF4vGdOixYtatB6o6OjMWnSpDrX79atGzIyMiCTyRq0PfJsGh31XVRUhFu3bqleJycnIyYmBiYmJrC3t8eMGTPw5ZdfwtnZWXV6lo2Njdq51s1JItTBm942+PNST3ThJwFXfge6TQPouBYhRAMyMjJUz7dv344FCxYgMTFRVWZgYKB6zhiDQqF47r2PAcDc3LxecQiFwmf2dJIXo9EW9aVLl+Dj4wMfHx8AwMyZM+Hj44MFCxYAAD7++GNMnToVkyZNgp+fH4qKinD48GHNnEP92Bg/e+xXvIISJgIeJgHpFzUWCyHk5WZlZaWaZDIZeDye6vWNGzdgaGiIQ4cOoUuXLhCJRDhz5gxu376NwYMHw9LSEgYGBvDz88OxY8fU1vvvrm8ej4eff/4ZQ4cOhUQigbOzM/bu3aua/++u76ou6iNHjsDNzQ0GBgbo16+f2g+LyspKTJs2DUZGRjA1NcWcOXMQGhpa74bYunXr0K5dOwiFQri4uOD3339XzWOMYdGiRbC3t4dIJIKNjQ2mTZummv9///d/cHZ2hp6eHiwtLTFixIh6bbu5aDRR9+zZE4yxGtOmTZsAcB+Ozz//HJmZmSgrK8OxY8fQoUMHTYYMT1sZHG0scUDRlSu4+ptG4yGENA3GGErKKzUyMcYa7X188skn+Prrr5GQkIBOnTqhqKgIAwYMwPHjx3H16lX069cPgwYNQlpa2jPXs3jxYowcORLXrl3DgAEDEBISgtzc3KfWLykpwYoVK/D777/j1KlTSEtLw6xZs1Tzly1bhs2bN2Pjxo2IiopCQUEBdu/eXa/3Fh4ejunTp+Ojjz7C9evX8d5772HcuHE4ceIEAOCvv/7Cd999hx9//BFJSUnYvXs3PD09AXANxWnTpuHzzz9HYmIiDh8+jFdffbVe228udK3vBhjtb48/9/TEWzqnwK6Hg9dvGSAyeO5yhJCWo7RCAfcFRzSy7fjPgyERNs7X8+eff44+ffqoXpuYmMDLy0v1+osvvkB4eDj27t2LKVOmPHU9Y8eOxZgxYwAAX331FVavXo2LFy+iX79+tdavqKjA+vXr0a4ddxXHKVOm4PPPqy8UtWbNGsydOxdDhw4FAPzwww84ePBgvd7bihUrMHbsWEyePBkA1yt7/vx5rFixAr169UJaWhqsrKwQFBQEXV1d2Nvbw9/fHwCQlpYGfX19vPHGGzA0NISDg4Oqd1fbaO1gMm022NsG13XccUdpBV5FMRC/W9MhEUJIrXx9fdVeFxUVYdasWXBzc4ORkREMDAyQkJDw3BZ1p06dVM/19fUhlUpVl8isjUQiUSVpgLuMZlX9/Px8ZGVlqZImAAgEAnTp0qVe7y0hIQGBgYFqZYGBgUhISAAAvPXWWygtLUXbtm0xceJEhIeHo7KyEgDQp08fODg4oG3btnj77bexefNmlJSU1Gv7zYVa1A0g1dPFG51ssCOmJ+bwtwHXdwE+/9V0WISQRiTWFSD+82CNbbux6Ovrq72eNWsWIiIisGLFCrRv3x5isRgjRoxAeXn5M9ejq6ur9prH40GpVNarfmN26deFnZ0dEhMTcezYMURERGDy5MlYvnw5IiMjYWhoiCtXruDkyZM4evQoFixYgEWLFiE6OlrrTgGjFnUDjfa3x05FD8xSTkH+4I2aDocQ0sh4PB4kQh2NTE15hbSoqCiMHTsWQ4cOhaenJ6ysrJCSktJk26uNTCaDpaUloqOjVWUKhQJXrlyp13rc3NwQFRWlVhYVFaV2oSuxWIxBgwZh9erVOHnyJM6dO4e4uDgAgI6ODoKCgvDNN9/g2rVrSElJwd9///0C76xpUIu6gTrbG8HY0g47s4zh9c8jvB1A5w8SQrSfs7Mzdu3ahUGDBoHH42H+/PnPbBk3lalTp2Lp0qVo3749XF1dsWbNGjx69KheP1Jmz56NkSNHwsfHB0FBQdi3bx927dqlGsW+adMmKBQKdO3aFRKJBH/88QfEYjEcHBywf/9+3LlzB6+++iqMjY1x8OBBKJVKuLi4NNVbbjBqUTcQj8fDaD97AMDWi+lgSiWggQ87IYTUx8qVK2FsbIxu3bph0KBBCA4ORufOnZs9jjlz5mDMmDF45513EBAQAAMDAwQHB9fr9NshQ4bg+++/x4oVK+Dh4YEff/wRGzduRM+ePQFw94P+6aefEBgYiE6dOuHYsWPYt28fTE1NYWRkhF27duH111+Hm5sb1q9fj61bt8LDw6OJ3nHD8VhzHzRoZnfv3oWdnR3S09Nha2vbqOvOKymH/1fHMYwdwyKzE9Dr/yXgOqBRt0EIaXplZWVITk6Gk5OTRq/T8DJTKpVwc3PDyJEj8cUXX2g6nEbxrM9VfXITtahfgJFEiP4dreDEy4Be/m26pCghhNRRamoqfvrpJ9y8eRNxcXH44IMPkJycjP/85z+aDk3rUKJ+QaP97LFV8ToWKiehuN8qTYdDCCEtAp/Px6ZNm+Dn54fAwEDExcXh2LFjcHNz03RoWocGk72gV9qagGfaHr/mWMM9sQij/Ew0HRIhhGg9Ozu7GiO2Se2oRf2CuEFldgC4QWUAgNZ92J8QQkgzokTdCIZ3sYWugAeX++EoW9ONbtRBCCGk0VCibgRmBiL0cbeEHz8Reg//Aa7+/vyFCCGEkDqgRN1IRvvZY3tlTwAA+ycckBdpNiBCCCGtAiXqRtK9vRkyZN7cjTrKi+hGHYQQQhoFJepGwufzMMrPHjsUPbmCK9T9TQgh5MVRom5Eb/naIVzZAwrGA9LPAzlJmg6JEEKeq2fPnpgxY4bqtaOjI1atWvXMZXg8Hnbv3v3C226s9TzLokWL4O3t3aTbaEqUqBuRlUwPHV1dcULpzRXQoDJCSBMaNGgQ+vXrV+u806dPg8fj4dq1a/Veb3R0NCZNmvSi4al5WrLMyMhA//79G3VbrQ0l6kY2xt9O1f3NYrYCigrNBkQIabUmTJiAiIgI3L17t8a8jRs3wtfXF506dar3es3NzSGRSBojxOeysrKCSCRqlm21VJSoG9lrHcxxXT8AD5gMvOJs4BLdq5oQ0jTeeOMNmJubY9OmTWrlRUVF2LFjByZMmICHDx9izJgxaNOmDSQSCTw9PbF169ZnrvffXd9JSUl49dVXoaenB3d3d0RERNRYZs6cOejQoQMkEgnatm2L+fPno6KCa6hs2rQJixcvRmxsLHg8Hng8nirmf3d9x8XF4fXXX4dYLIapqSkmTZqEoqLqs2jGjh2LIUOGYMWKFbC2toapqSnCwsJU26oLpVKJzz//HLa2thCJRPD29sbhw4dV88vLyzFlyhRYW1tDT08PDg4OWLp0KQCAMYZFixbB3t4eIpEINjY2mDZtWp233RB0CdFGpiPgY7ifI76PHIYvdTcCJ5YAniMACV1alJAWqby4/ssIRIDg8derohJQyAEeH9AVP3+9Qv06b0ZHRwfvvPMONm3ahHnz5qnu5bxjxw4oFAqMGTMGRUVF6NKlC+bMmQOpVIoDBw7g7bffRrt27eDv7//cbSiVSgwbNgyWlpa4cOEC8vPz1Y5nVzE0NMSmTZtgY2ODuLg4TJw4EYaGhvj4448xatQoXL9+HYcPH1bdK1omk9VYR3FxMYKDgxEQEIDo6GhkZ2fj3XffxZQpU9R+jJw4cQLW1tY4ceIEbt26hVGjRsHb2xsTJ06s0377/vvv8e233+LHH3+Ej48PfvnlF7z55pv4559/4OzsjNWrV2Pv3r34888/YW9vj/T0dKSnc1ee/Ouvv/Ddd99h27Zt8PDwQGZmJmJjY+u03YaiRN0ERvrZoeeJ1xGiPAa3snTg5FJgwHJNh0UIaYivbOq/zFubAI+h3PMb+4AdYwGH7sC4A9V1VnkCJQ9rLrsov16bGj9+PJYvX47IyEjVfZg3btyI4cOHQyaTQSaTYdasWar6U6dOxZEjR/Dnn3/WKVEfO3YMN27cwJEjR2Bjw+2Lr776qsZx5c8++0z13NHREbNmzcK2bdvw8ccfQywWw8DAADo6OrCysnrqtrZs2YKysjL89ttv0NfnfrD88MMPGDRoEJYtWwZLS0sAgLGxMX744QcIBAK4urpi4MCBOH78eJ0T9YoVKzBnzhyMHj0aALBs2TKcOHECq1atwtq1a5GWlgZnZ2d0794dPB4PDg4OqmXT0tJgZWWFoKAg6Orqwt7evk778UVQ13cTsDWWoLe7DT6vfIcriP4fkJ2g2aAIIa2Sq6srunXrhl9++QUAcOvWLZw+fRoTJkwAACgUCnzxxRfw9PSEiYkJDAwMcOTIEaSlpdVp/QkJCbCzs1MlaQAICAioUW/79u0IDAyElZUVDAwM8Nlnn9V5G09uy8vLS5WkASAwMBBKpRKJiYmqMg8PDwgEAtVra2trZGdn12kbBQUFuH//PgIDA9XKAwMDkZDAfU+PHTsWMTExcHFxwbRp03D06FFVvbfeegulpaVo27YtJk6ciPDwcFRWVtbrfdYXtaibyEd9XRAcn4VwRSACfP1hZeTw/IUIIdrn0/v1X0bwxOAo10HcOnj/ahfNiHuxuJ4wYcIETJ06FWvXrsXGjRvRrl07vPbaawCA5cuX4/vvv8eqVavg6ekJfX19zJgxA+Xl5Y22/XPnziEkJASLFy9GcHAwZDIZtm3bhm+//bbRtvEkXV1dtdc8Hg9KpbLR1t+5c2ckJyfj0KFDOHbsGEaOHImgoCDs3LkTdnZ2SExMxLFjxxAREYHJkyerejT+HVdj0eoWtUKhwPz58+Hk5ASxWIx27drhiy++AGsBd6fqYGmIoT5t8GHFZMx60B8QNs8ISkJIIxPq138SPNEGEuhwZU8en37Wehtg5MiR4PP52LJlC3777TeMHz9edbw6KioKgwcPxn//+194eXmhbdu2uHnzZp3X7ebmhvT0dGRkZKjKzp8/r1bn7NmzcHBwwLx58+Dr6wtnZ2ekpqaqv12hEAqF4rnbio2NRXFx9fH7qKgo8Pl8uLi41DnmZ5FKpbCxsalxi82oqCi4u7ur1Rs1ahR++uknbN++HX/99Rdyc3MBAGKxGIMGDcLq1atx8uRJnDt3DnFxjffD69+0ukW9bNkyrFu3Dr/++is8PDxw6dIljBs3DjKZrMlH2TWGD4M6YF/sfZy5lYOzt3LQra0xwJSAoGl+dRFCXk4GBgYYNWoU5s6di4KCAowdO1Y1z9nZGTt37sTZs2dhbGyMlStXIisrSy0pPUtQUBA6dOiA0NBQLF++HAUFBZg3b55aHWdnZ6SlpWHbtm3w8/PDgQMHEB4erlbH0dERycnJiImJga2tLQwNDWuclhUSEoKFCxciNDQUixYtwoMHDzB16lS8/fbbquPTjWH27NlYuHAh2rVrB29vb2zcuBExMTHYvHkzAGDlypWwtraGj48P+Hw+duzYASsrKxgZGWHTpk1QKBTo2rUrJBIJ/vjjD4jFYrXj2I1Nq1vUZ8+exeDBgzFw4EA4OjpixIgR6Nu3Ly5ebBm3kbQzkSCkK/fH271/N9hPvYBzazUcFSGkNZowYQIePXqE4OBgtePJn332GTp37ozg4GD07NkTVlZWGDJkSJ3Xy+fzER4ejtLSUvj7++Pdd9/FkiVL1Oq8+eab+PDDDzFlyhR4e3vj7NmzmD9/vlqd4cOHo1+/fujVqxfMzc1rPUVMIpHgyJEjyM3NhZ+fH0aMGIHevXvjhx9+qN/OeI5p06Zh5syZ+Oijj+Dp6YnDhw9j7969cHZ2BsCNYP/mm2/g6+sLPz8/pKSk4ODBg+Dz+TAyMsJPP/2EwMBAdOrUCceOHcO+fftgamraqDE+ice0uB/5q6++woYNG3D06FF06NABsbGx6Nu3L1auXImQkJBal5HL5ZDL5arX9+7dg7u7O9LT02Fra9tcoas8KJTjteUn0K/yBFYK1wPSNsC0GEBH2OyxEEJqV1ZWhuTkZDg5OUFPT0/T4ZBW4lmfq7t378LOzq5OuUmru74/+eQTFBQUwNXVFQKBAAqFAkuWLHlqkgaApUuXYvHixc0Y5bOZG4owPtAJa09UoKOoCKETF0JASZoQQkgdaXXX959//onNmzdjy5YtuHLlCn799VesWLECv/7661OXmTt3LvLz81VTfHx8M0Zcu4mvtoVULMLn+QMQfrPxRloSQghp/bQ6Uc+ePRuffPIJRo8eDU9PT7z99tv48MMPVZdyq41IJIJUKlVNhoaGzRhx7WRiXXzQsx0A4LuIm5BXKoCUKEB7jzoQQgjRElqdqEtKSsDnq4coEAga9Xy55hIa4AhLqQj38kqQsWEksGkAcP0vTYdFCCFEy2l1oh40aBCWLFmCAwcOICUlBeHh4Vi5ciWGDh2q6dDqTSwUYFpvZwA8HMox4wojFjTsOsKEEEJeGlqdqNesWYMRI0Zg8uTJcHNzw6xZs/Dee+/hiy++0HRoDTLS1w6OphKsKumHApE1UHAPiFqt6bAIIY+1xN46or0a6/Ok1adnNYb6DIFvDnti7mH6thgME13CSt5KQEcPmHIJMLLTdGiEvLSUSiWSkpIgEAhgbm4OoVCourIXIfXFGEN5eTkePHgAhUIBZ2fnGodxW83pWa3RoE42WB95B7syumC6uQ8cCq9yXeBv0X2rCdEUPp8PJycnZGRk4P79Blzbm5BaSCQS2Nvb10jS9UWJupnx+TzMDu6A8ZsuYdqjUditGwveP7sA/4mAQzdNh0fIS0soFMLe3h6VlZXPvSY1Ic8jEAigo6PTKD0zlKg1oJeLBfwcjRGdAkRbvAH/3L3AoTnAu8cAHdFzlyeENA0ejwddXd0muwsSIQ2h1YPJWisej4eP+7kCAKZkDoRSKAUyrwFbRtEocEIIIWooUWuIn6MJermYI1tpiLUWiwBdfeDOCeD3oUBpnqbDI4QQoiUoUWvQrGDu/qrf3rLCnQGbAT0ZkH4B2PQGUPRAw9ERQgjRBpSoNcjDRoZBXtzt6L6I0QfGHgT0LQCmAPgCDUdHCCFEG1Ci1rCZfTpAwOfhROIDnMizAMYfBt4OByQmmg6NEEKIFqBErWFOZvoYH+gIAPg0PA6F+vaAoVV1hSu/A5lxmgmOEEKIxlGi1gIz+7jAwVSCjPwyLD10o3rGjQPA3inApoFAXrrmAiSEEKIxlKi1gFgowNfDOgEAtlxIw7nbD7kZDoGAXVfAawwg0/zlTwkhhDQ/StRaIqCdKUK62gMAPtl1DaXlCkBsBLyzBwheClRd3aZ1X5qdEELIv1Ci1iKf9HeFjUwPqQ9L8O3RRK5QVwxUXSe2Us5dFOXyr4CSLnFICCEvA0rUWsRQTxdLhnkCAP4XlYwraY/UK1z9A0g6AuybBnzrChz8GEi7QK1sQghpxShRa5leLhYY5tMGjAEf77wGeeUTLecu44CecwGxMVCcDVz8EfilL7CqExCxEMi4RkmbEEIaW6UcyL0DPLytkc1TotZC899wh5mBELeyi/DD37eqZ/D5QM9PgFlJwH92AJ1GAUIDID8NiFoF/NgDWOsPnFwG5Nx66voJIeSlJy8C7l8Fru1Qb+BEfQ9sHADE7awuu3cZWO0D7Jve/HGC7p6llYz1hfhicEd8sPkK1p28jf4dreFuI62uINAFOvTlpopS4OYR4PpO4OZRIOcmcPIrbrL2Avp9XX37THkhUJABGJhzrXJCCGnNGAOKHwAPErnvxqrpwU2g4G51Pec+3OBdgGs5p0YBTq9Wz5eYcvdjEGjmrmqUqLVUf09r9O9ohUPXMzF7Zyx2hwVCV1BLB4iuGPAYwk1lBdy519f/Am7/DWTEcudfOzyumxIFbB0FWHsD70VWryP8A+6ypUL9x78sGffIlE88rypXcs99QoC2Pbnls28AZ1cD0jbA6/Oq13tiKfdPwhcAAiHA1+E+6Hxd7rG259ZegKU7t3xZPpB8GtDVA9oHVa/3/lXuR4fIEDBy4H50NMI9XwkhzUyp5P53G+v/t+gBELuFS8Q5N4GcRO575Gn0zQFTZ0BRUV3m/V/A6TXAyrO6zNwFmHe/cWJsAErUWmzxYA+cvf0Q/9wvwIZTdxDWq/2zF9CTAt5juKn4IZCwh0t8VSrLuBt/GFioL5ewDygvrF9wDt2qE3VhBhCzGbDsqJ6o43YAufU8ptPrs+pE/SgV2B4CGFgCs25W1zk0h7t5SRWhAWBkzyVtI/uaEyVyQppPRWn1j3IAuB8D3DoGmLYDPIZyZWUFwJrOQMlDQEcMGFoChtbcVRmrHg2s1F+LDNS3k7AfiPuT+x7yHc+VlRcBEQv+FRAPMHYAzFwAM2cu6Zp14KbaLtVs58dNWoQStRazMNTDgjfc8dGOWHx/PAnBHlZob2Hw/AUBQN+0+sNbparlrVRWlzEGDFjODU4rLwF4/MdJjffEL11ezXI7/+p1mDgBvRcC+mbq2+v6HlCSCygrAWUFoKh6fDwp//WoqODWVUVXwl3wRfyvfyZjR+5XcukjoCiL++fMjuem2ggNgaCFgP9E7nX2Da5nQd8CeDeiut6f73C9EBJTrtfBxhuw8QHMXTXW5UWIVmCM+58rygIKM2t/LMoCCrMAeT4w7lD1Ibe70cDfXwBug6oTtciQu50vUwIVxVx3c+6dZ8egqw9MPA5YuHGvc28D8XsAnqD6u87IHvB8CzBpB5h34JKzaXuuV64Fo0St5YZ1boO9sfcRefMB5vx1DX++FwAB/wVbh/wnutB5PK4F/iKMHYEeM2uWd33vxdZr1h6YcLRm+bAN1c8rSoH8u0BeKpCXpj49Sn38A6RQvftLUQ48SuFGcj6pIIMrf5TCDR6pIhABVh25pG3t/UTypn8f0kIxxvWE6RkBQglXdvcycPMQ1xXsNYorKy8B1nThWr4K+VNXV0PxE7fptezIdSfbdqku4/GA909z268oeZzkM7ikr3p8Yiov5BL6g8TqRN3udS5J2/pWr5cvAIb/3JA9otV4jLXu83nu3r0LOzs7pKenw9a2ZV6G815eKfqujERxuQILB7ljXKDT8xcinPISID+d+wUvtXlcVgxkxXOJ1sanum5WPNc6z0/nuuvuX+VOeZPXcoxLR4/7Auq/rPqL4vYJruVg5QkM+r667qpOQFE294Vo5MD9sDGuenTkymS21GonTaM0D8hOALL/4T7jVb1PZflA6H7AqQdXL/pn4MBHgOsbwOjNXBljwBfmXK8XAIhkXDe1geXj7mnLJ7qpLblHAwvuEFtjHm6SF3L/QwYW3P9yK1Cf3ERNghagjZEYcwe44bPd1/HN4UQEuVnCzkSi6bBaBqGEOyalVqZf+zGoqmPjdv5Ax+Hcc6USeJTMJe37V7mu8YxYQF4A3LvEtb6rErW8gGuJC4Tq660oBSofTyUPgftXam6bJwBkbaoTt9+E6h8RxQ+5bkEDCy7BE+2gqOB+1D16ojfH7Y0n/m453Hm3hpbc37XRtlvJtULBuIRYJfM69yP0UYp6Ui64V/t6eAKufhUrL8BvInfIR1WHB0w6AYik3MAroYa+d0SGrSZBN4TWJ+p79+5hzpw5OHToEEpKStC+fXts3LgRvr6+z1+4FfmPvz32xd7HheRcTPztErztjB7/YOVxh5Lx+JDy49fcHKC9hQHG+NtDp7YR4+T5+HxuEIxpO8BzBFemVHKJMyMGsOpUXdeuKzBmG/eF9qQJR7gvRXnB46711Oou9rxU7rVCXv1lD3Cj3Ku+8FNOATvGcjdpGXewer0/9eaO/+ub1xyMUzUQx8Ci9bfUK0q51qFQv3G/zJVKLslV/Y3y0qoPsTxKBQrvPz4z4gk2PtV/tzsngb8mAI49gLH7q+ts6Mm1VAW6j8dvKB5PldzE/vVaqQDeWFl9fPfmIWD7f7nP25OHhv4YDhRl1v5epLbcD1GLx5OlOzeYSkdUXedpg6ieHP1MNEKrE/WjR48QGBiIXr164dChQzA3N0dSUhKMjV++c4D5fB6WDe+Eft+fwo3MQtzIrPso7X3XMrBmjA8spS17QIXW4PO54+dm/xqFb2gFuPSvWd+kbfXz2r70lEruGJ0qcadwX6YqPG6QjLRNdRFjQOY17nj7M/G4QX5VCfzVj6u/jO9dBi5t5Hocuk2tXuTvL7nk9+S2VKfqKWt/7TGEO2YIcAMI085xg/WeNXq2Us5ddKK8kOvalBdxhx7kj1+XF3E/TqpaeJlxwJFPufWO+F/1etZ3Bx4+vsCPxLT6kMK/J2kb7hjmvxXnAHcvcYdCqk4DVCqApXbccdFn0RGrn2FQdfwU4AZgGjsCMjv1fZlxjUvG9fFky7eqx+bfYyyM7LjEK22jnpQt3KrPESYtklYfo/7kk08QFRWF06dPN3gdreEY9ZMup+bi/J1cMMa470s8/t5E9WswBgagvFKJzRfSUCSvhKm+EKtGe6OHs/mzN0BaBsa40bQludzAnapBOGqDcjJrJoSRvwHug7nncTtrb/Etc+RG1NfHwJVcdz0A3IkEfnuTG3E75WJ1nfXdgfzH3bDywurjns/S98vqHxF3LwE/9wZk9sCHcdV1NvTiDkvgOV9lfF0umRo7AK/NAexf4cqv7wJ2jgNs/dXPAljViWtRy+y4ZapO/6s6PGFkz/VY1OdYrFLJHTIpyeX+Nnwd7scDT/D4uU51Gf9xGU8ASK2ru7kVj1vaAqH6wFDSorSaY9R79+5FcHAw3nrrLURGRqJNmzaYPHkyJk6cqOnQNKaLgwm6ONRy7t9TjPa3x+TNV5CQUYB3frmIqb3aY3pQhxcaOX7tbh4eFpfDz9EEBiKt/gi1Xv8+Ra42SiV3TPzJ0bSWHavnW3oAr89Xb/EBQNcPuNZs1XZ4fFSfrlf1/N+n63WtXl5HBLTxrXk8vfghUJpbM05dCXcuvMiA67oWGnLPhQbcqTVVTNsBw37mTj180riD3OA+eWF1j0SNKZX7YZB7m5vaB1UnajNnwNITsHBVX++7x7hTAxtzdD+f//y/2/MIdOiMg5eMVreo9fS4rtqZM2firbfeQnR0NKZPn47169cjNDS01mXkcjnk8uouoXv37sHd3b3VtKgboqxCgcX74rH1Inf8M6CtKb4f4w0Lw7p3hSuVDMdvZGN95G1cTuVaWzp8Hjo7GOO1DuZ41dkcHjZS8F/01DHSehVkcC11Hu9xYjbkHpsj6SgV3A+VqsRt4Qa06fK8pQhpMvVpUWt1ohYKhfD19cXZs2dVZdOmTUN0dDTOnTtX6zKLFi3C4sWLa5S/zIm6yu6r9/BpeBxKyhUwMxBh9WhvdGtv9sxlyiuV2Bt7Hz9G3kZSNtfKEgr4sJCKcPdRqVpdU30hujub4VVnc/ToYFavHwKEEPIyafJEnZ6eDh6Pp1r5xYsXsWXLFri7u2PSpEkNi7oWDg4O6NOnD37+ufoE9nXr1uHLL7/EvXu1n3JALepnu5VdhLDNV5CYVQg+D5jeuwOmvN6+Rld4sbwSWy+m4X9nkpGRXwYAMBDpIOQVe4wPdIKlVA+pD4tx6uYDRN7MwbnbOSguVz8e6mYtxasdzNDHzRJdHIzBo8t4EkIIgGZI1D169MCkSZPw9ttvIzMzEy4uLvDw8EBSUhKmTp2KBQv+fa3VhvnPf/6D9PR0tcFkH374IS5cuKDWyn6W1jaYrDGUliuwYM917LjM3T2me3szrBrtDTMDEXKK5NgUlYLfz6civ5Qb7GNuKML4QCeEvGIPqV7tp/qUVypxJe0RTt18gNNJOYi7p36RkA6WBvjvKw4Y6tMGhk9ZByGEvCyaPFEbGxvj/PnzcHFxwerVq7F9+3ZERUXh6NGjeP/993HnznOu2VpH0dHR6NatGxYvXoyRI0fi4sWLmDhxIjZs2ICQkJA6rYMS9dPtvHwXn+2OQ1mFEhaGIvRyscDumHuQV3LnhjqZ6WPSq20x1KcN9HRrOa3lGR4WyXHmVg5OJj7A4euZKK3gWtsSoQBDfNrgv10d1G/dSQghL5EmT9QGBga4fv06HB0d8eabbyIwMBBz5sxBWloaXFxcUFpa+vyV1NH+/fsxd+5cJCUlwcnJCTNnzqzXqG9K1M92M6sQkzdfwa3Hx58BwMtWhvdfa4e+HlYvfl1xAAVlFdh1+S7+uJCmtp0uDsb47yv26N/Rus4/BPJLK5CcU4yUnGIIdfgIaGsKY33h8xckhBAt0uSJumvXrujVqxcGDhyIvn374vz58/Dy8sL58+cxYsQI3L179/kraSaUqJ+vWF6JZYdvIKugDKHdHBHQ1rRJjiczxnD+Ti7+uJCKI9czUankPnom+kK85WuLEH8H2JtKUFJeiZScEqQ8LEZyTvWUklOMh8XqF/jg8YBObWTo4WyOHs5m8LE3hlCHzi0lhGi3Jk/UJ0+exNChQ1FQUIDQ0FD88ssvAIBPP/0UN27cwK5duxoWeROgRK2dsgvKsC06HVsvpqkGq/F4gLmBCNmFz75Lj6VUBEdTfeSXVtS4Qpu+UICAdqZ4tYM5ejibw9FUQoPYCCFap1lOz1IoFCgoKFC7nGdKSgokEgksLCwassomQYlau1UqlDh+Ixt/nE/F6aQcVblMrIu25vpwMtWHk5k+nMz14fj4uf4TF1nJKijD6aQcnE56gDNJOTVa3LbGYvRwNsdAT2t0d372qWiEENJcmjxRl5aWgjEGiYS7k0pqairCw8Ph5uaG4ODghkXdRChRtxzpuSXIKZLD0VS/QcedlUqG+IwCnE7KwambD3ApNRcViuqP9yAvGywa5A5TA9Ez1kIIIU2vyRN13759MWzYMLz//vvIy8uDq6srdHV1kZOTg5UrV+KDDz5ocPCNjRL1y6ukvBIX7uTiaHwWtkenQcm44+GL3/TAG52sqUucEKIx9clNDRp1c+XKFfTowd1sfOfOnbC0tERqaip+++03rF69uiGrJKTRSYQ66OVqgaXDPLE7LBCuVobILS7H1K1X8f4fl5FdUKbpEAkh5LkalKhLSkpgaMjd9/Xo0aMYNmwY+Hw+XnnlFaSmpjZqgIQ0hk62Rtg7pTum93aGDp+HI/9koc93p/DX5bvQ4qvoEkJIwxJ1+/btsXv3bqSnp+PIkSPo27cvACA7OxtSKV3EgmgnoQ4fH/bpgH1Tu6NjGynySyvw0Y5YjNsUjft5jXfuPyGENKYGJeoFCxZg1qxZcHR0hL+/PwICAgBwrWsfH59GDZCQxuZmLcXuyYH4uJ8LhDp8nEx8gL7fncKWC2nUuiaEaJ0Gn56VmZmJjIwMeHl5gf/45uUXL16EVCqFq6vrc5ZuPjSYjDzLrexCzN55DVfT8gAA3dqZ4pP+rrCS6kEq1q33pVMJIaQumvU2l1VXIdPWJEiJmjyPQsmw6WwKlh+5gbIKpdo8PV0+ZGJdGImFkIl1IZPoPn7NPXZsI0NPF3MaQU4IqZf65KYG3bFdqVTiyy+/xLfffouiIu7azYaGhvjoo48wb948VQubkJZAwOdhQncn9Ha1wOf743E17RHySyugZEBZhRJlFXJkFTz9ammd7Y3w6QA3+DqaNGPUhJCXRYMS9bx58/C///0PX3/9NQIDAwEAZ86cwaJFi1BWVoYlS5Y0apCENAdHM338MtYPAHfxlKLySuSXVCC/tAJ5VY+l5cgvrUB+SQUeFMlxKC4TV9LyMGL9OQR7WOLjfq5oZ26g4XdCCGlNGtT1bWNjg/Xr1+PNN99UK9+zZw8mT56Me/fuNVqAL4q6vklTyi4ow3fHklQXVBHweRjtZ4fpQc6wMNTTdHiEEC3V5Bc8yc3NrXXAmKurK3JzcxuySkJaJAupHpYO88SRGa8iyM0SCiXD5gtp6Ln8JFYdu4lieaWmQySEtHANStReXl744YcfapT/8MMP6NSp0wsHRUhL42xpiJ9DfbF90ivwsjNCSbkCq44l4bXlJ7H5QioqFcrnr4QQQmrRoK7vyMhIDBw4EPb29qpzqM+dO4f09HQcPHhQdXlRbUBd36S5McZwMC4T3xy5gdSHJQCAdub6CO3miDZGYpgbimBhqAczAyF0BDTwkpCXUbOcnnX//n2sXbsWN27cAAC4ublh0qRJ+PLLL7Fhw4aGrLJJUKImmlJeqcSWC6lY/fct5P7r9psAd/9tU30hzAxEsJDqwcJQpJp8HU3QsY1MA1ETQppDs55H/aTY2Fh07twZCoWisVb5wihRE00rKKvApqgUxKbnIbtQjuzCMuQUlUOhfPq/Ho8HvPdqO8zs0wFCHWp1E9LaNPl51ISQupPq6WJab2e1MqWSIbekHNkFXOLOLpTjweMpOacYkTcfYH3kbUTdysH3o73Rlk75IuSlRYmaEA3g83kwMxDBzEAEd9S8kc3h6xmY81cc4u7lY+DqM1j0pjtG+trRFdAIeQlRnxohWqhfR2scntED3dqZorRCgTl/xWHy5ivIK6l5rJsQ0rrVq0U9bNiwZ87Py8t7kVgIIU+wlonxx4Su2HD6DlYcScSh65m4mpaHlaO80K2dmabDI4Q0k3olapns2aNQZTIZ3nnnnRcKiBBSjc/n4f3X2iGwnRmmb7uKOznFCPn5At5/rR0+DKKBZoS8DBp11Lc2olHfpLUoKa/E5/visS06HQDQyVaG70f7wMlMX8OREULqq8kvIaopX3/9NXg8HmbMmKHpUAhpdhKhDr4e3gnrQjpDJtbFtbv5GLj6NP66fFfToRFCmlCLSdTR0dH48ccf6RKl5KXX35MbaBbQ1hQl5Qp8tCMWe2K050Y4hJDG1SISdVFREUJCQvDTTz/B2NhY0+EQonHWMjH+eLcrxnZzBADM3nENF+481GxQhJAm0SISdVhYGAYOHIigoKDn1pXL5SgoKFBNhYWFzRAhIc1PwOdhwRvu6OdhhXKFEhN/u4Rb2fR5J6S10fpEvW3bNly5cgVLly6tU/2lS5dCJpOpJnd39yaOkBDN4fN5WDXaGz72Rigoq8TYjdF4UCjXdFiEkEak1Yk6PT0d06dPx+bNm6Gnp1enZebOnYv8/HzVFB8f38RREqJZeroC/PyOLxxNJbj7qBQTfo1GSTndB5uQ1kKrE/Xly5eRnZ2Nzp07Q0dHBzo6OoiMjMTq1auho6NT680/RCIRpFKpajI0NNRA5IQ0L1MDETaN84exhBsNPm3r1Wfe9ONZyiuV2B6dhmPxWY0cJSGkIbQ6Uffu3RtxcXGIiYlRTb6+vggJCUFMTAwEAoGmQyREazia6ePnUD+IdPg4lpCNxfv+QX0vk3D2Vg4GrD6NOX/FYeLvl3DuNg1QI0TTtPqmHIaGhujYsaNamb6+PkxNTWuUE0KALg7GWDXKG5O3XMFv51JhZyzBxFfbPne5zPwyLDmYgH2x9wFwA9UUSoaZf8bg8PRXIZPoNnXohJCn0OoWNSGk/vp7WmPeADcAwJKDCThwLeOpdSsUSvx8+g56f3sS+2Lvg88D3glwwJk5veBkpo+M/DJ8Gh5X75Y5IaTxaHWLujYnT57UdAiEaL0J3Z1w91EpNp1NwYd/xsBSKoKvo4lanfN3HmLBnuu4mVUEAPC2M8KXQzqiYxvumv6rRnlj+LqzOBCXgV5XLDCiC12ClxBNoBY1Ia0Qj8fD/Dfc0cfdEuWVSrz72yXcecAl5OzCMny4PQajN5zHzawiGEt0sWy4J3Z90E2VpAHAy84IH/bpAABYuOc6Uh8Wa+S9EPKyo0RNSCsl4POwerQPvOyMkFdSgbEbo7E+8jZ6r4hE+NV74PGAkK72ODGrJ0b52YPP59VYx/uvtYO/kwmKyxWYsT0GlQqlBt4JIS83StSEtGJioQD/C/WFvYkEabkl+PrQDRTKK9HJVobdkwOxZKgnjCTCpy4v4POwcqQXDPV0cDUtD2v+vtWM0RNCAErUhLR6ZgYibBznBzMDEYwkuvhqqCfCJwfCy86oTsvbGkuwZKgnAGDN30m4nJrbhNESQv6txQ0mI4TUXztzA5z6uCcEfB5EOvW//sCbXjY4cSMb4VfvYcb2GByc1gOGenTKFiHNgVrUhLwkJEKdBiXpKosHe8DWWIz03FIs3PtPI0ZGCHkWStSEkDqR6uli1Shv8HnAriv3VBdHIYQ0LUrUhJA683U0wZRe7QEA88LjcC+vVMMREdL6UaImhNTL1N7O8Lbjbqs5c3tMg2/+QQipG0rUhJB60RXw8f1ob+gLBbiQnIsfT93WdEiEtGqUqAkh9eZgqo+Fb3oAAFYevYnY9DzNBkRIK0anZxFCGuStLrY4mZiNg3GZGLw2Cnq6fBhLhJCJdWEsEcJYXxdGEiGMJbowEgthJOHKLaQi2BpLYCzRBY9X82pohBB1lKgJIQ3C4/Hw1VBP3H1Uimt381FWoURGfhky8svqtLy+UABbYwlsjcWPJ4naoxElckIAUKImhLwAI4kQe8ICUSSvRF5JBR6VlD/18VFJBfJKypGZX4bsQjmKyxVIzCpEYlZhres2EOnglbYmmNnHBe420mZ+Z4RoD0rUhJAXwuPxYKinC0M9XdiZSOq0TFmFAvfzSnH3USnSH5Xg7qPSxxP3/EGhHEXyShxLyMbxG9kY7GWDj/q61Hn9hLQmlKgJIc1OT1eAtuYGaGtuUOv8sgoFbj8owo+Rd7A39j52x9zHgbgMhHR1wNTX28PUQNTMEROiOTTqmxCidfR0BfCwkWH1GB/sn9odPZzNUKFg2HQ2Ba8tP4nVx5NQLK/UdJiENAtK1IQQrdaxjQy/T+iKPyZ0hWcbGYrklVgZcROvLT+J38+loILukU1aOUrUhJAWobuzGfaEBWLNGB84mEqQUyTH/D3/oM/KSOyLvQ8lXSGNtFKUqAkhLQafz8MgLxtEfPgavhjsATMDEVIelmDq1qvo//1p/H4uBYVlFZoOk5BGxWOMteqfoXfv3oWdnR3S09Nha2ur6XAIIY2oWF6J/51JxoZTd1D0+Ji1RCjAYG8bhHR1QMc2Mg1HSEjt6pObKFETQlq8/NIKhF+5i80X0pCUXaQq97KVIaSrA97wsoZESCe5EO1BifoJlKgJeXkwxhCd8gibL6TiUFwmyh8PNDPU08Hwzrb4T1d7dLA01HCUhNQvN2n1MeqlS5fCz88PhoaGsLCwwJAhQ5CYmKjpsAghWorH48HfyQTfj/bBubmvY25/VziYSlBYVolNZ1PQ97tTGLn+HA5fz0Arb6OQVkSrE3VkZCTCwsJw/vx5REREoKKiAn379kVxcbGmQyOEaDlTAxHee60dTnzUE79P8Ec/DysI+DxcTMnF+39cwcgfzyHubr6mwyTkuVpU1/eDBw9gYWGByMhIvPrqq3Vahrq+CSFVsgrK8Pu5VPx85g7KKpTg8YDhnW0xO9gFllI9TYdHXiKtpuv73/LzuV+/JiYmT60jl8tRUFCgmgoLa7/gPyHk5WMp1cOsYBecmNUTQ33agDFg5+W76LXiJH74OwllFQpNh0hIDS0mUSuVSsyYMQOBgYHo2LHjU+stXboUMplMNbm7uzdjlISQlsBaJsZ3o7wRPrkbfOyNUFKuwIqjN9H7W+7iKS2oo5G8BFpM1/cHH3yAQ4cO4cyZM8/sJpDL5ZDL5arX9+7dg7u7O3V9E0JqxRjD3tj7WHboBu4/vpd2FwdjLHjDHV52RpoNjrRara7re8qUKdi/fz9OnDjx3DckEokglUpVk6EhnYpBCHk6Ho+Hwd5tcPyjnpjZpwPEugJcTn2EwWujMPPPGKTnlmg6RPKS0+orADDGMHXqVISHh+PkyZNwcnLSdEiEkFZKLBRgWm9njPS1wzdHbmDXlXuqycNGij7ulujjbgl3ayl4PJ6mwyUvEa3u+p48eTK2bNmCPXv2wMXFRVUuk8kgFovrtA4a9U0IaYjY9DwsP5KIqNs5ePJbso2RWJW0/Z1MoCtoER2TRMu0miuTPe1X68aNGzF27Ng6rYMSNSHkRTwskuP4jWxExGfhdNIDlFVU31ZTqqeDXq4W6ONuidc6mMNQT1eDkZKWpD65Seu7vgkhRJNMDUQY6WuHkb52KC1X4MytHETEZ+J4QjYeFpdjT8x97Im5D10BD33cLRHWqz08bOhmIKTxaHWiJoQQbSIWClTd3golw5W0RzgWn4WI+CzcySnGwbhMHIzLRB93S0x73RmetpSwyYvT6q7vxkBd34SQ5pCQUYB1J29j37X7qmPar7taYOrr7eFjb6zZ4IjWaXWnZxFCiLZzs5Zi9RgfRHz4Gob5tAGfB/x9IxtD/+8s3vnlIi6n5mo6RNJCUaImhJBG1N7CACtHeeP4Rz0xoostBHweTt18gOHrziHk5/O4cOehpkMkLQwdoyaEkCbgZKaPFW95Ydrrzvi/k7ew8/JdRN16iKhbD9HVyQTDOreBrbEENkZiWMv0oKcr0HTIREtRoiaEkCZkbyrB18M7Ycrr7bHu5G38eSkdF5JzcSFZvSvczEAIGyMxbGRitDEWw8ZIjDZGerAxEsNKqgdTAxEEfLrQysuIEjUhhDQDW2MJlgz1xJTX22PT2RQkZBTifl4p7j0qRWmFAjlF5cgpKse1p9wjm8/jThWzMKya9GApFcFcqlddJtWDtVQPfErorQolakIIaUbWMjHm9ndTvWaMIa+kAvfySnH/8cQ9L8O9x89ziuRQMuBBoRwPCuX45xnrb2MkxrhAR4z0s4OULsDSKlCiJoQQDeLxeDDWF8JYX4iObWo/77pSocTD4nJkF8iRXViG7EK5+vNCOR4UlOFBkRz38krx5YEErDqWhLd8bTGumxPsTSXN/K5IY6JETQghWk5HwIelVA+WUj0AT7+ISlmFAuFX7+GXM8lIyi7CxqgU/Ho2BX3cLTGhe1v4ORrTDUVaIErUhBDSSujpCjDG3x6j/exwKikH/zuTjFM3H+DIP1k48k8WPNvIMKG7EwZ4WkOoQ2fnthR0ZTJCCGnFkrIK8UtUMnZduQd5JXdDEUupCO8EOGJY5zawltXtToSkcbWau2c1BkrUhBDC3QVsy4U0/HY+FQ8K5apyS6kIXrZG8LIzgredETxtZTQIrRlQon4CJWpCCKkmr1Rgf2wGfj+firh7+VAoa6aAdub6quTtZWcEN2tDiHTogiyNqdXc5pIQQkjjEukIMLyLLYZ3sUVJeSX+uV+A2PQ8xKTn4drdfKTlluD2g2LcflCMXVfvAQB0BTy0tzBEB0sDOFsYoL2FIZwtDeBgIoGOgI51NzVK1IQQ8pKSCHXg52gCP0cTVVlucTli7+YhNv3xdDcfucXlSMgoQEJGgdryQgEfbc310d7CAM5VidzSAI6m+pTAGxElakIIISom+kL0crFALxcLANwFWe4+KkViZiGSsouQlMU93souQmmFAjcyC3EjsxBAhmodUj0dBLY3Qw9nc/RwNoOdCZ3H/SIoURNCCHkqHo8HOxMJ7EwkCHK3VJUrlQz38kqRlF2IpKwiLolnF+FWViEKyipx6HomDl3PBMDdoKSHM5e4A9qZwkD0/NSjUDJkFpQh7WEJ0nNLcPdRCcwMuYFvbtbSl+r0MkrUhBBC6o3Pr07gr7tWJ3CFkuHa3TycTsrB6aQHuJKWh+ScYiTnFOO3c6nQ4fPQ2d4Yr3YwQ3dnc+gKeEjPLUGaaipVJeYKRe1jnYU6fHjYSOFlawQfeyN42RrBwVTSai/mQqO+CSGENJmCsgqcv/0Qp5Ie4HRSDlIfltR5WV0BD7bG3I+BNkZ6uJ9Xhti7ecgrqahR10iiCy9b7hQzb3sjWEn1INLhQ6Qr4B51+BDq8CEU8LUiodOob0IIIVpBqqeLvh5W6OthBQBIfVisam2fu/0QOgI+7EwksDeRwN5E/PhRH/amElhJ9Wrc2pMxhtSHJYh5PFI9Jj0P8fcLkFdSgcibDxB588FzY6pK3FVJ3EiiCwvD6ruQ/fuOZGYGQo2enkYtakIIIS2avFKBGxmFiFGNVM9Dfmkl5JUKyCuVKH98RbYXwSVzEdytpVg12ueF10ctakIIIS8NkY5AdXGW2iiVDOUKJeSVSsgrFSivfPy8QomySgVyi8rxoOjpdySrUHC3Is0rqYBYt/lb1i0iUa9duxbLly9HZmYmvLy8sGbNGvj7+2s6LEIIIS0An8+DHl8APV0BgPpdHrXqfuFc8i6DQAPHt7V+fPv27dsxc+ZMLFy4EFeuXIGXlxeCg4ORnZ2t6dAIIYS0clX3C3exMkQPZ3N0a2/W7DFofaJeuXIlJk6ciHHjxsHd3R3r16+HRCLBL7/8ounQCCGEkCan1Ym6vLwcly9fRlBQkKqMz+cjKCgI586d02BkhBBCSPPQ6mPUOTk5UCgUsLS0VCu3tLTEjRs3al1GLpdDLq++hVthYWGTxkgIIYQ0Ja1uUTfE0qVLIZPJVJO7u7umQyKEEEIaTKsTtZmZGQQCAbKystTKs7KyYGVlVesyc+fORX5+vmqKj49vjlAJIYSQJqHVXd9CoRBdunTB8ePHMWTIEACAUqnE8ePHMWXKlFqXEYlEEIlEqtd5eXkAgIyMjFrrE0IIIc2tKicplc+/GItWJ2oAmDlzJkJDQ+Hr6wt/f3+sWrUKxcXFGDduXJ2Wr2qN03nXhBBCtE1WVhbs7e2fWUfrE/WoUaPw4MEDLFiwAJmZmfD29sbhw4drDDB7Gh8fH1y8eBGWlpbg81+sp7+wsBDu7u6Ij4+HoaHhC63rZUH7rP5on9Uf7bP6o31Wf425z5RKJbKysuDj8/zLkbb6a303poKCAshkMuTn50MqlWo6nBaB9ln90T6rP9pn9Uf7rP40tc+0ejAZIYQQ8rKjRE0IIYRoMUrU9SASibBw4UK1UeXk2Wif1R/ts/qjfVZ/tM/qT1P7jI5RE0IIIVqMWtSEEEKIFqNETQghhGgxStSEEEKIFqNEXQ9r166Fo6Mj9PT00LVrV1y8eFHTIWmtpUuXws/PD4aGhrCwsMCQIUOQmJio6bBajK+//ho8Hg8zZszQdCha7d69e/jvf/8LU1NTiMVieHp64tKlS5oOS2spFArMnz8fTk5OEIvFaNeuHb744gvQUCV1p06dwqBBg2BjYwMej4fdu3erzWeMYcGCBbC2toZYLEZQUBCSkpKaLB5K1HW0fft2zJw5EwsXLsSVK1fg5eWF4OBgZGdnazo0rRQZGYmwsDCcP38eERERqKioQN++fVFcXKzp0LRedHQ0fvzxR3Tq1EnToWi1R48eITAwELq6ujh06BDi4+Px7bffwtjYWNOhaa1ly5Zh3bp1+OGHH5CQkIBly5bhm2++wZo1azQdmlYpLi6Gl5cX1q5dW+v8b775BqtXr8b69etx4cIF6OvrIzg4GGVlZU0TECN14u/vz8LCwlSvFQoFs7GxYUuXLtVgVC1HdnY2A8AiIyM1HYpWKywsZM7OziwiIoK99tprbPr06ZoOSWvNmTOHde/eXdNhtCgDBw5k48ePVysbNmwYCwkJ0VBE2g8ACw8PV71WKpXMysqKLV++XFWWl5fHRCIR27p1a5PEQC3qOigvL8fly5cRFBSkKuPz+QgKCsK5c+c0GFnLkZ+fDwAwMTHRcCTaLSwsDAMHDlT7rJHa7d27F76+vnjrrbdgYWEBHx8f/PTTT5oOS6t169YNx48fx82bNwEAsbGxOHPmDPr376/hyFqO5ORkZGZmqv2PymQydO3atcnygdbflEMb5OTkQKFQ1LgRiKWlJW7cuKGhqFoOpVKJGTNmIDAwEB07dtR0OFpr27ZtuHLlCqKjozUdSotw584drFu3DjNnzsSnn36K6OhoTJs2DUKhEKGhoZoOTyt98sknKCgogKurKwQCARQKBZYsWYKQkBBNh9ZiZGZmAkCt+aBqXmOjRE2aXFhYGK5fv44zZ85oOhStlZ6ejunTpyMiIgJ6enqaDqdFUCqV8PX1xVdffQWAu1Pe9evXsX79ekrUT/Hnn39i8+bN2LJlCzw8PBATE4MZM2bAxsaG9pkWo67vOjAzM4NAIFDd27pKVlYWrKysNBRVyzBlyhTs378fJ06cgK2trabD0VqXL19GdnY2OnfuDB0dHejo6CAyMhKrV6+Gjo4OFAqFpkPUOtbW1nB3d1crc3NzQ1pamoYi0n6zZ8/GJ598gtGjR8PT0xNvv/02PvzwQyxdulTTobUYVd/5zZkPKFHXgVAoRJcuXXD8+HFVmVKpxPHjxxEQEKDByLQXYwxTpkxBeHg4/v77bzg5OWk6JK3Wu3dvxMXFISYmRjX5+voiJCQEMTExEAgEmg5R6wQGBtY45e/mzZtwcHDQUETar6SkBHy++te+QCCAUqnUUEQtj5OTE6ysrNTyQUFBAS5cuNBk+YC6vuto5syZCA0Nha+vL/z9/bFq1SoUFxdj3Lhxmg5NK4WFhWHLli3Ys2cPDA0NVcduZDIZxGKxhqPTPoaGhjWO3+vr68PU1JSO6z/Fhx9+iG7duuGrr77CyJEjcfHiRWzYsAEbNmzQdGhaa9CgQViyZAns7e3h4eGBq1evYuXKlRg/frymQ9MqRUVFuHXrlup1cnIyYmJiYGJiAnt7e8yYMQNffvklnJ2d4eTkhPnz58PGxgZDhgxpmoCaZCx5K7VmzRpmb2/PhEIh8/f3Z+fPn9d0SFoLQK3Txo0bNR1ai0GnZz3fvn37WMeOHZlIJGKurq5sw4YNmg5JqxUUFLDp06cze3t7pqenx9q2bcvmzZvH5HK5pkPTKidOnKj1+ys0NJQxxp2iNX/+fGZpaclEIhHr3bs3S0xMbLJ46O5ZhBBCiBajY9SEEEKIFqNETQghhGgxStSEEEKIFqNETQghhGgxStSEEEKIFqNETQghhGgxStSEEEKIFqNETQghhGgxStSEkEbH4/Gwe/duTYdBSKtAiZqQVmbs2LHg8Xg1pn79+mk6NEJIA9BNOQhphfr164eNGzeqlYlEIg1FQwh5EdSiJqQVEolEsLKyUpuMjY0BcN3S69atQ//+/SEWi9G2bVvs3LlTbfm4uDi8/vrrEIvFMDU1xaRJk1BUVKRW55dffoGHhwdEIhGsra0xZcoUtfk5OTkYOnQoJBIJnJ2dsXfvXtW8R48eISQkBObm5hCLxXB2dq7xw4IQwqFETchLaP78+Rg+fDhiY2MREhKC0aNHIyEhAQBQXFyM4OBgGBsbIzo6Gjt27MCxY8fUEvG6desQFhaGSZMmIS4uDnv37kX79u3VtrF48WKMHDkS165dw4ABAxASEoLc3FzV9uPj43Ho0CEkJCRg3bp1MDMza74dQEhL0mT35SKEaERoaCgTCARMX19fbVqyZAljjLsF6fvvv6+2TNeuXdkHH3zAGGNsw4YNzNjYmBUVFanmHzhwgPH5fJaZmckYY8zGxobNmzfvqTEAYJ999pnqdVFREQPADh06xBhjbNCgQWzcuHGN84YJaeXoGDUhrVCvXr2wbt06tTITExPV84CAALV5AQEBiImJAQAkJCTAy8sL+vr6qvmBgYFQKpVITEwEj8fD/fv30bt372fG0KlTJ9VzfX19SKVSZGdnAwA++OADDB8+HFeuXEHfvn0xZMgQdOvWrUHvlZDWjhI1Ia2Qvr5+ja7oxiIWi+tUT1dXV+01j8eDUqkEAPTv3x+pqak4ePAgIiIi0Lt3b4SFhWHFihWNHi8hLR0doybkJXT+/Pkar93c3AAAbm5uiI2NRXFxsWp+VFQU+Hw+XFxcYGhoCEdHRxw/fvyFYjA3N0doaCj++OMPrFq1Chs2bHih9RHSWlGLmpBWSC6XIzMzU61MR0dHNWBrx44d8PX1Rffu3bF582ZcvHgR//vf/wAAISEhWLhwIUJDQ7Fo0SI8ePAAU6dOxdtvvw1LS0sAwKJFi/D+++/DwsIC/fv3R2FhIaKiojB16tQ6xbdgwQJ06dIFHh4ekMvl2L9/v+qHAiFEHSVqQlqhw4cPw9raWq3MxcUFN27cAMCNyN62bRsmT54Ma2trbN26Fe7u7gAAiUSCI0eOYPr06fDz84NEIsHw4cOxcuVK1bpCQ0NRVlaG7777DrNmzYKZmRlGjBhR5/iEQiHmzp2LlJQUiMVi9OjRA9u2bWuEd05I68NjjDFNB0EIaT48Hg/h4eEYMmSIpkMhhNQBHaMmhBBCtBglakIIIUSL0TFqQl4ydLSLkJaFWtSEEEKIFqNETQghhGgxStSEEEKIFqNETQghhGgxStSEEEKIFqNETQghhGgxStSEEEKIFqNETQghhGgxStSEEEKIFvt/+bYG7Ts5FBgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.ticker import MaxNLocator\n",
    "\n",
    "\n",
    "def plot_losses(epochs_seen, tokens_seen, train_losses, val_losses):\n",
    "    fig, ax1 = plt.subplots(figsize=(5, 3))\n",
    "\n",
    "    ax1.plot(epochs_seen, train_losses, label=\"Training loss\")\n",
    "    ax1.plot(epochs_seen, val_losses, linestyle=\"-.\", label=\"Validation loss\")\n",
    "    ax1.set_xlabel(\"Epochs\")\n",
    "    ax1.set_ylabel(\"Loss\")\n",
    "    ax1.legend(loc=\"upper right\")\n",
    "    ax1.xaxis.set_major_locator(MaxNLocator(integer=True))\n",
    "\n",
    "    ax2 = ax1.twiny()\n",
    "    ax2.plot(tokens_seen, train_losses, alpha=0) \n",
    "    ax2.set_xlabel(\"Tokens seen\")\n",
    "\n",
    "    fig.tight_layout()\n",
    "    plt.savefig(\"loss-plot.pdf\")\n",
    "    plt.show()\n",
    "\n",
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78c99a9",
   "metadata": {},
   "source": [
    "## 控制随机性的解码策略​​\n",
    "\n",
    "- 对于像我们上面训练的GPT这样的相对较小的大语言模型（LLM），其推理过程的计算成本相对较低。因此，即使您在上面训练时使用了GPU，在进行推理时也​​无需使用GPU​​；\n",
    "- 利用我们之前在简单训练函数中使用过的 generate_text_simple函数（来自前一章），我们可以​​逐词（或逐词元）地生成新文本​​；\n",
    "- 如上所述，下一个生成的词元是​​词汇表中所有词元里对应概率得分最高的那个​​。这种方法也称为贪心解码（Greedy Decoding），它在每一步都选择最可能的词元，但可能导致生成文本缺乏多样性；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "9b471186",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort moves you?\"\n",
      "\n",
      "\"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\"\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.to(\"cpu\")\n",
    "model.eval()\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"Every effort moves you\", tokenizer),\n",
    "    max_new_tokens=25,\n",
    "    context_size=GPT_CONFIG_124M[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e4c2e85",
   "metadata": {},
   "source": [
    "- 即使我们多次执行上述的generate_text_simple函数，大语言模型（LLM）也​​总是会生成相同的输出结果​​；\n",
    "- 我们现在引入两个概念（即所谓的​​解码策略​​）来修改generate_text_simple函数：​​温度缩放​​与​​Top-K采样​​；\n",
    "- ​​这些策略使得模型能够​​控制生成文本的随机性与多样性；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7f38ffb",
   "metadata": {},
   "source": [
    "### 温度缩放"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac7d1f9b",
   "metadata": {},
   "source": [
    "- 之前，我们始终通过torch.argmax函数​​选取概率最高的词元​​作为下一个输出；\n",
    "- 为增加文本的多样性，我们可以改用torch.multinomial(probs, num_samples=1)函数来​​从概率分布中进行采样​​，从而生成下一个词元；\n",
    "- ​​每个词元索引被选中的几率，与其在输入张量中的概率值成正比​​；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "ae907cbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = { \n",
    "    \"closer\": 0,\n",
    "    \"every\": 1, \n",
    "    \"effort\": 2, \n",
    "    \"forward\": 3,\n",
    "    \"inches\": 4,\n",
    "    \"moves\": 5, \n",
    "    \"pizza\": 6,\n",
    "    \"toward\": 7,\n",
    "    \"you\": 8,\n",
    "} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "c3fa45de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forward\n"
     ]
    }
   ],
   "source": [
    "inverse_vocab = {v: k for k, v in vocab.items()}\n",
    "\n",
    "next_token_logits = torch.tensor(\n",
    "    [4.51, 0.89, -1.90, 6.75, 1.63, -1.62, -1.89, 6.28, 1.79]\n",
    ")\n",
    "\n",
    "probas = torch.softmax(next_token_logits, dim=0)\n",
    "next_token_id = torch.argmax(probas).item()\n",
    "\n",
    "print(inverse_vocab[next_token_id])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "d232ee29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([6.0907e-02, 1.6313e-03, 1.0019e-04, 5.7212e-01, 3.4190e-03, 1.3257e-04,\n",
       "        1.0120e-04, 3.5758e-01, 4.0122e-03])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7c624f4e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'closer',\n",
       " 1: 'every',\n",
       " 2: 'effort',\n",
       " 3: 'forward',\n",
       " 4: 'inches',\n",
       " 5: 'moves',\n",
       " 6: 'pizza',\n",
       " 7: 'toward',\n",
       " 8: 'you'}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inverse_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fc378ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forward\n"
     ]
    }
   ],
   "source": [
    "next_token_id = torch.multinomial(probas, num_samples=1).item()\n",
    "print(inverse_vocab[next_token_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd8555d",
   "metadata": {},
   "source": [
    "- 我们不再通过torch.argmax确定最可能的词元，而是使用torch.multinomial(probas, num_samples=1)​​从softmax概率分布中采样​​来确定下一个词元；\n",
    "- 为直观演示，我们将观察使用原始softmax概率对下一个词元​​采样1000次​​的结果分布；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "27b520b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "73 x closer\n",
      "0 x every\n",
      "0 x effort\n",
      "582 x forward\n",
      "2 x inches\n",
      "0 x moves\n",
      "0 x pizza\n",
      "343 x toward\n",
      "0 x you\n"
     ]
    }
   ],
   "source": [
    "def print_sampled_tokens(probas):\n",
    "    torch.manual_seed(123) # Manual seed for reproducibility\n",
    "    sample = [torch.multinomial(probas, num_samples=1).item() for i in range(1000)]\n",
    "    sampled_ids = torch.bincount(torch.tensor(sample), minlength=len(probas))\n",
    "    for i, freq in enumerate(sampled_ids):\n",
    "        print(f\"{freq} x {inverse_vocab[i]}\")\n",
    "\n",
    "print_sampled_tokens(probas)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8647aa93",
   "metadata": {},
   "source": [
    "- 我们可以通过一种称为​​温度缩放​​的概念来控制概率分布和选择过程；\n",
    "- \"温度缩放\"只是一个专业术语，其本质是​​将逻辑值除以一个大于0的数​​；\n",
    "- ​​温度大于1​​会导致在应用softmax函数后，词元概率分布​​更趋于均匀​​；\n",
    "- 温度小于1​​会导致在应用softmax函数后，概率分布​​更具确定性（更尖锐或更集中）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "7adea537",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax_with_temperature(logits, temperature):\n",
    "    scaled_logits = logits / temperature\n",
    "    return torch.softmax(scaled_logits, dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "874707e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "temperatures = [1, 0.1, 5]\n",
    "\n",
    "scaled_probas = [softmax_with_temperature(next_token_logits, T) for T in temperatures]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "652d5606",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(scaled_probas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "fb393b35",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([6.0907e-02, 1.6313e-03, 1.0019e-04, 5.7212e-01, 3.4190e-03, 1.3257e-04,\n",
       "        1.0120e-04, 3.5758e-01, 4.0122e-03])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_probas[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "8cd1d2b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([6.0907e-02, 1.6313e-03, 1.0019e-04, 5.7212e-01, 3.4190e-03, 1.3257e-04,\n",
       "        1.0120e-04, 3.5758e-01, 4.0122e-03])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "7a9a8fc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.8530e-10, 3.5189e-26, 2.6890e-38, 9.9099e-01, 5.7569e-23, 4.4220e-37,\n",
       "        2.9718e-38, 9.0133e-03, 2.8514e-22])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_probas[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "1c38ff75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.1546, 0.0750, 0.0429, 0.2421, 0.0869, 0.0454, 0.0430, 0.2203, 0.0898])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_probas[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "f2f16887",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAATOZJREFUeJzt3XlcVNX/P/DXsINsIpsgCoomFDtKuKFFghpqpBlqKCLfLHGBcI1FIMA0Ef2EYirua0ZamibyEXHNHTMRA0RIQXElQNY5vz/8cT+OA8h+7+D7+XjM48OcuXfmNfOZfM8999xzRIwxBkIIIYQIkhzfAQghhBBSPyrUhBBCiIBRoSaEEEIEjAo1IYQQImBUqAkhhBABo0JNCCGECBgVakIIIUTAqFATQgghAqbAd4D2JhaLce/ePWhoaEAkEvEdhxBCyBuIMYZ///0XRkZGkJNr+Jj5jSvU9+7dg4mJCd8xCCGEEOTn56Nbt24NbvPGFWoNDQ0ALz4cTU1NntMQQgh5ExUXF8PExISrSQ154wp1bXe3pqYmFWpCCCG8aswpWBpMRgghhAgYr4U6LS0NHh4eMDIygkgkwv79+1+7T2pqKuzt7aGsrAxzc3Ns3ry5zXMSQgghfOG1UJeWlsLGxgbx8fGN2v727dsYNWoUhg0bhqtXr2Lu3LmYPn06fv/99zZOSgghhPCD13PUI0aMwIgRIxq9fUJCAszMzLBixQoAgIWFBU6dOoWVK1fCzc2trWISQtqZWCxGZWUl3zEIaTZFRUXIy8u3ynPJ1GCys2fPwtXVVaLNzc0Nc+fOrXefiooKVFRUcPeLi4vbKh4hpBVUVlbi9u3bEIvFfEchpEW0tbVhaGjY4jk7ZKpQFxYWwsDAQKLNwMAAxcXFeP78OVRVVaX2iYmJQXh4eHtFJIS0AGMMBQUFkJeXh4mJyWsngiBEiBhjKCsrw4MHDwAAXbt2bdHzyVShbo5FixYhMDCQu1977RohRHiqq6tRVlYGIyMjqKmp8R2HkGarPXB88OAB9PX1W9QNLlOF2tDQEPfv35dou3//PjQ1Nes8mgYAZWVlKCsrt0c8QhpviVYDjz1rvxwCU1NTAwBQUlLiOQkhLVf7Y7OqqqpFhVqm+pWcnZ2RkpIi0ZacnAxnZ2eeEhFC2gLNw086gtb6HvNaqEtKSnD16lVcvXoVwIvLr65evYq8vDwAL7qtvb29ue1nzJiBnJwczJ8/Hzdv3sSaNWuwd+9eBAQE8BGfEEIIaXO8FuqLFy/Czs4OdnZ2AIDAwEDY2dkhNDQUAFBQUMAVbQAwMzPDoUOHkJycDBsbG6xYsQIbNmygS7MIIYR0WLyeox46dCgYY/U+XtesY0OHDsWVK1faMBUhRGhMFx5q19fLXTqq0du+rnszLCwMS5YsaWEiYTE1NcXcuXMbvDRW6GbPno3Tp0/j+vXrsLCw4Hp2hUimBpMRQojQFBQUcH/v2bMHoaGhyMzM5NrU1dX5iNVkjDHU1NRAQaH9ykJlZSWvAwenTZuGP/74A9euXeMtQ2PI1GAyQggRGkNDQ+6mpaUFkUgk0bZ7925YWFhARUUFffv2xZo1a7h9c3NzIRKJsHfvXgwePBiqqqro168fbt26hQsXLsDR0RHq6uoYMWIEioqKuP2mTp2KsWPHIjw8HHp6etDU1MSMGTMkZnMTi8WIiYmBmZkZVFVVYWNjg3379nGPp6amQiQS4fDhw3BwcICysjJOnTqF7OxsjBkzBgYGBlBXV0e/fv1w7Ngxbr+hQ4fizp07CAgIgEgk4noUlixZAltbW4nPJi4uDqamplK5o6KiYGRkhLfeegvAi2WHP/nkE2hra0NHRwdjxoxBbm5ua/zfU6/Vq1dj5syZ6NmzZ5u+TmugQk0IIW1kx44dCA0NRVRUFDIyMhAdHY2QkBBs2bJFYruwsDAEBwfj8uXLUFBQwMSJEzF//nysWrUKJ0+eRFZWFjd2p1ZKSgoyMjKQmpqKXbt2ISkpSWJyp5iYGGzduhUJCQn466+/EBAQgMmTJ+PEiRMSz7Nw4UIsXboUGRkZsLa2RklJCUaOHImUlBRcuXIF7u7u8PDw4MYLJSUloVu3boiIiEBBQYFEj0JjpKSkIDMzE8nJyTh48CCqqqrg5uYGDQ0NnDx5EqdPn4a6ujrc3d0bnEZWXV29wduMGTOalEvIqOubEELaSFhYGFasWAFPT08ALwbE3rhxA+vWrcOUKVO47YKCgrhBsXPmzIGXlxdSUlIwcOBAAICvr6/UmB0lJSUkJiZCTU0Nb7/9NiIiIjBv3jxERkaiqqoK0dHROHbsGHf5as+ePXHq1CmsW7cOLi4u3PNERETggw8+4O7r6OjAxsaGux8ZGYmff/4Zv/zyC/z9/aGjowN5eXloaGjA0NCwyZ9Jp06dsGHDBq7Le/v27RCLxdiwYQN3dL5p0yZoa2sjNTUVw4cPr/N5XndOWVNTs8nZhIoKNSGEtIHS0lJkZ2fD19cXfn5+XHt1dTW0tCQnvLG2tub+rp0m2crKSqKtdjrKWjY2NhKztzk7O6OkpAT5+fkoKSlBWVmZRAEGXpwTrr3Kppajo6PE/ZKSEixZsgSHDh1CQUEBqqur8fz5c4krcFrCyspK4rx0eno6srKyoKGhIbFdeXk5srOz630ec3PzVskjC6hQE0JIGygpKQEArF+/Hk5OThKPvTpLlaKiIvd37VHlq21NWaSk9rUPHToEY2NjicdenamxU6dOEveDgoKQnJyM7777Dubm5lBVVcW4ceNeu5qZnJyc1FU8VVVVUtu9+nolJSVwcHDAjh07pLbV09Or9/VeN0hv8uTJSEhIaHAbWUGFmhBC2oCBgQGMjIyQk5ODSZMmtfrzp6enSyxGdO7cOairq8PExAQ6OjpQVlZGXl6eRDd3Y5w+fRpTp07FRx99BOBFIX11YJeSkhI33WstPT09FBYWgjHG/dhozCVP9vb22LNnD/T19ZvUXU1d34QQQlosPDwcs2fPhpaWFtzd3VFRUYGLFy/iyZMnEosFNUdlZSV8fX0RHByM3NxchIWFwd/fH3JyctDQ0EBQUBACAgIgFosxaNAgPHv2DKdPn4ampqbE+fFX9e7dG0lJSfDw8IBIJEJISIjU0bypqSnS0tLw6aefQllZGbq6uhg6dCiKioqwbNkyjBs3DkeOHMHhw4dfWzAnTZqE5cuXY8yYMYiIiEC3bt1w584dJCUlYf78+ejWrVud+7W06zsrKwslJSUoLCzE8+fPucJvaWkpuLnmadQ3IYS0kenTp2PDhg3YtGkTrKys4OLigs2bN8PMzKzFz/3++++jd+/eGDJkCCZMmIDRo0dLTKwSGRmJkJAQxMTEwMLCAu7u7jh06NBrXzs2NhadO3fGgAED4OHhATc3N9jb20tsExERgdzcXPTq1YvrnrawsMCaNWsQHx8PGxsbnD9/HkFBQa99H2pqakhLS0P37t3h6ekJCwsL+Pr6ory8vE2PiqdPnw47OzusW7cOt27d4mbJvHfvXpu9ZnOJWENTg3VAxcXF0NLSwrNnzzpU1wiRMbR6Vp3Ky8tx+/ZtmJmZQUVFhe84gjV16lQ8ffoU+/fv5zsKaUBD3+em1CI6oiaEEEIEjAo1IYQQImA0mIwQQmRMXQsWkY6LjqgJIYQQAaNCTQghhAgYFWpCCCFEwKhQE0IIIQJGhZoQQggRMCrUhBBCiIBRoSaEkBYQiUQN3l6e1rOjMDU1RVxcHN8xWiQvLw+jRo2Cmpoa9PX1MW/ePFRXVze4T1RUFAYMGAA1NTVoa2u3T1DQddSEEFnQ0JSrbfJ6jZ/GtaCggPt7z549CA0NRWZmJtf2uuUYhYIxhpqaGigotF9ZqKys5GUBjJqaGowaNQqGhoY4c+YMCgoK4O3tDUVFRURHR9e7X2VlJcaPHw9nZ2ds3Lix3fLSETUhhLSAoaEhd9PS0oJIJJJo2717NywsLKCiooK+fftizZo13L65ubkQiUTYu3cvBg8eDFVVVfTr1w+3bt3ChQsX4OjoCHV1dYwYMQJFRUXcflOnTsXYsWMRHh4OPT09aGpqYsaMGRJrRovFYsTExMDMzAyqqqqwsbHBvn37uMdTU1MhEolw+PBhODg4QFlZGadOnUJ2djbGjBkDAwMDqKuro1+/fjh27Bi339ChQ3Hnzh0EBARwvQYAsGTJEtja2kp8NnFxcTA1NZXKHRUVBSMjI7z11lsAgPz8fHzyySfQ1taGjo4OxowZI7W0Zms6evQobty4ge3bt8PW1hYjRoxAZGQk4uPjG1x3Ozw8HAEBAbCysmqzbHWhQk0IIW1kx44dCA0NRVRUFDIyMhAdHY2QkBBs2bJFYruwsDAEBwfj8uXLUFBQwMSJEzF//nysWrUKJ0+eRFZWFkJDQyX2SUlJQUZGBlJTU7Fr1y4kJSUhPDycezwmJgZbt25FQkIC/vrrLwQEBGDy5Mk4ceKExPMsXLgQS5cuRUZGBqytrVFSUoKRI0ciJSUFV65cgbu7Ozw8PJCXlwcASEpKQrdu3RAREYGCggKJHoXGSElJQWZmJpKTk3Hw4EFUVVXBzc0NGhoaOHnyJE6fPg11dXW4u7s3WDTV1dUbvM2YMaPefc+ePQsrKysYGBhwbW5ubiguLsZff/3VpPfTHqjrmxBC2khYWBhWrFgBT09PAICZmRlu3LiBdevWSawJHRQUBDc3NwDAnDlz4OXlhZSUFAwcOBAA4OvrKzVtqJKSEhITE6Gmpoa3334bERERmDdvHiIjI1FVVYXo6GgcO3YMzs7OAICePXvi1KlTWLduHVxcXLjniYiIwAcffMDd19HRgY2NDXc/MjISP//8M3755Rf4+/tDR0cH8vLy0NDQgKGhYZM/k06dOmHDhg1cl/f27dshFouxYcMG7uh806ZN0NbWRmpqKoYPH17n89SuH12fhlakKiwslCjSALj7hYWFjX0r7YYKNSGEtIHS0lJkZ2fD19cXfn5+XHt1dTW0tCTPuVtbW3N/1xaMl7tXDQwM8ODBA4l9bGxsoKamxt13dnZGSUkJ8vPzUVJSgrKyMokCDLw4x2pnZyfR5ujoKHG/pKQES5YswaFDh1BQUIDq6mo8f/6cO6JuKSsrK4nz0unp6cjKyoKGhobEduXl5cjOzq73eczNzVsljyygQk0IIW2gpKQEALB+/Xo4OTlJPCYvLy9xX1FRkfu79qjy1TaxWNzk1z506BCMjY0lHlNWVpa436lTJ4n7QUFBSE5OxnfffQdzc3Ooqqpi3LhxDXZDA4CcnBwYYxJtVVVVUtu9+nolJSVwcHDAjh07pLbV09Or9/VeN0hv8uTJSEhIqPMxQ0NDnD9/XqLt/v373GNCQ4WaEELagIGBAYyMjJCTk4NJkya1+vOnp6fj+fPnUFVVBQCcO3cO6urqMDExgY6ODpSVlZGXlyfRzd0Yp0+fxtSpU/HRRx8BeFFIXx3YpaSkhJqaGok2PT09FBYWgjHG/dh4Xfc0ANjb22PPnj3Q19dvsLv6VS3p+nZ2dkZUVBQePHgAfX19AEBycjI0NTVhaWnZ6AzthQo1IYS0kfDwcMyePRtaWlpwd3dHRUUFLl68iCdPniAwMLBFz11ZWQlfX18EBwcjNzcXYWFh8Pf3h5ycHDQ0NBAUFISAgACIxWIMGjQIz549w+nTp6GpqSlxfvxVvXv3RlJSEjw8PCASiRASEiJ1NG9qaoq0tDR8+umnUFZWhq6uLoYOHYqioiIsW7YM48aNw5EjR3D48OHXFt9JkyZh+fLlGDNmDCIiItCtWzfcuXMHSUlJmD9/Prp161bnfi3p+h4+fDgsLS3x2WefYdmyZSgsLERwcDBmzpzJ9TicP38e3t7eSElJ4Xol8vLy8PjxY+Tl5aGmpob7sWBubt6ml+HxPuo7Pj4epqamUFFRgZOTk1R3xKvi4uLw1ltvQVVVFSYmJggICEB5eXk7pSWEkMabPn06NmzYgE2bNsHKygouLi7YvHkzzMzMWvzc77//Pnr37o0hQ4ZgwoQJGD16tMTkKpGRkQgJCUFMTAwsLCzg7u6OQ4cOvfa1Y2Nj0blzZwwYMAAeHh5wc3ODvb29xDYRERHIzc1Fr169uO5pCwsLrFmzBvHx8bCxscH58+cRFBT02vehpqaGtLQ0dO/eHZ6enrCwsICvry/Ky8ubdITdFPLy8jh48CDk5eXh7OyMyZMnw9vbGxEREdw2ZWVlyMzMlOi+Dw0NhZ2dHcLCwlBSUgI7OzvY2dnh4sWLbZKzloi9elKhHe3Zswfe3t5ISEiAk5MT4uLi8OOPPyIzM5PrjnjZzp07MW3aNCQmJmLAgAG4desWpk6dik8//RSxsbGNes3i4mJoaWnh2bNnbfYlIOS1GprAowmTbXQ05eXluH37NszMzKCiosJ3HMGaOnUqnj59iv379/MdhTSgoe9zU2oRr0fUsbGx8PPzg4+PDywtLZGQkAA1NTUkJibWuf2ZM2cwcOBATJw4Eaamphg+fDi8vLxeexROCCGEyCreCnVlZSUuXboEV1fX/4WRk4OrqyvOnj1b5z4DBgzApUuXuMKck5OD3377DSNHjmyXzIQQQkh7420w2cOHD1FTU1PnRec3b96sc5+JEyfi4cOHGDRoEBhjqK6uxowZM7B48eJ6X6eiogIVFRXc/eLi4tZ5A4QQwpNXJz8hHRvvg8maIjU1FdHR0VizZg0uX76MpKQkHDp0CJGRkfXuExMTAy0tLe5mYmLSjokJIYSQluHtiFpXVxfy8vLcRea17t+/X+8F5yEhIfjss88wffp0AC9muCktLcX//d//4euvv4acnPTvjkWLFklcBlFcXEzFmhBCiMzg7YhaSUkJDg4OSElJ4drEYjFSUlK4uWlfVVZWJlWMa2f4qW/wurKyMjQ1NSVuhBBCiKzgdcKTwMBATJkyBY6Ojujfvz/i4uJQWloKHx8fAIC3tzeMjY0RExMDAPDw8EBsbCzs7Ozg5OSErKwshISEwMPDQ2pKPkIIIaQj4LVQT5gwAUVFRQgNDUVhYSFsbW1x5MgRboBZXl6exBF0cHAwRCIRgoODcffuXejp6cHDwwNRUVF8vQVCCCGkTfE64QkfaMITIgg04UmdaMIT0pF0iAlPCCGEENIwKtSEENICIpGowdvL8293FKampoiLi+M7RovU9f/V7t27+Y5VJ1o9ixAieFZbrNr19f6c8mejty0oKOD+3rNnD0JDQ5GZmcm1teWqSq2JMYaamhooKLRfWaisrISSklK7vd6rNm3aBHd3d+6+trY2b1kaQkfUhBDSAoaGhtxNS0sLIpFIom337t2wsLCAiooK+vbtizVr1nD75ubmQiQSYe/evRg8eDBUVVXRr18/3Lp1CxcuXICjoyPU1dUxYsQIFBUVcftNnToVY8eORXh4OPT09KCpqYkZM2agsrKS20YsFiMmJgZmZmZQVVWFjY0N9u3bxz2empoKkUiEw4cPw8HBAcrKyjh16hSys7MxZswYGBgYQF1dHf369cOxY8e4/YYOHYo7d+4gICCAOxIFgCVLlsDW1lbis4mLi4OpqalU7qioKBgZGeGtt94CAOTn5+OTTz6BtrY2dHR0MGbMGKk1sNuCtra2xP9XQh0XQYWaEELayI4dOxAaGoqoqChkZGQgOjoaISEh2LJli8R2YWFhCA4OxuXLl6GgoICJEydi/vz5WLVqFU6ePImsrCyEhoZK7JOSkoKMjAykpqZi165dSEpKQnh4OPd4TEwMtm7dioSEBPz1118ICAjA5MmTceLECYnnWbhwIZYuXYqMjAxYW1ujpKQEI0eOREpKCq5cuQJ3d3d4eHggLy8PAJCUlIRu3bohIiICBQUFEj0KjZGSkoLMzEwkJyfj4MGDqKqqgpubGzQ0NHDy5EmcPn0a6urqcHd3l/jh8Sp1dfUGbzNmzHhtlpkzZ0JXVxf9+/dHYmJivfNx8I26vgkhpI2EhYVhxYoV8PT0BACYmZnhxo0bWLduHaZMmcJtFxQUBDc3NwDAnDlz4OXlhZSUFAwcOBAA4OvrKzW/t5KSEhITE6Gmpoa3334bERERmDdvHiIjI1FVVYXo6GgcO3aMm0CqZ8+eOHXqFNatWwcXFxfueSIiIvDBBx9w93V0dGBjY8Pdj4yMxM8//4xffvkF/v7+0NHRgby8PDQ0NOqdRbIhnTp1woYNG7gu7+3bt0MsFmPDhg3c0fmmTZugra2N1NRUDB8+vM7nuXr1aoOv87qR1BEREXjvvfegpqaGo0eP4ssvv0RJSQlmz57d5PfU1qhQE0JIGygtLUV2djZ8fX3h5+fHtVdXV0NLS/LyPGtra+7v2nkkrKysJNoePHggsY+NjQ3U1NS4+87OzigpKUF+fj5KSkpQVlYmUYCBF+eE7ezsJNocHR0l7peUlGDJkiU4dOgQCgoKUF1djefPn3NH1C1lZWUlcV46PT0dWVlZ0NDQkNiuvLwc2dnZ9T6Publ5i3KEhIRwf9vZ2aG0tBTLly+nQk0IIW+KkpISAMD69evh5OQk8dirMykqKipyf9ceVb7aJhaLm/zahw4dgrGxscRjysrKEvc7deokcT8oKAjJycn47rvvYG5uDlVVVYwbN67BbmjgxTLFr3YdV1VVSW336uuVlJTAwcEBO3bskNpWT0+v3td73SC9yZMnIyEhocFtXubk5ITIyEhUVFRIfUZ8o0JNCCFtwMDAAEZGRsjJycGkSZNa/fnT09Px/PlzqKqqAgDOnTsHdXV1mJiYQEdHB8rKysjLy5Po5m6M06dPY+rUqfjoo48AvCikrw7sUlJSQk1NjUSbnp4eCgsLwRjjfmy8rnsaAOzt7bFnzx7o6+s3aRKqlnZ91/V8nTt3FlyRBqhQE0JImwkPD8fs2bOhpaUFd3d3VFRU4OLFi3jy5InEqn7NUVlZCV9fXwQHByM3NxdhYWHw9/eHnJwcNDQ0EBQUhICAAIjFYgwaNAjPnj3D6dOnoampKXF+/FW9e/dGUlISPDw8IBKJEBISInU0b2pqirS0NHz66adQVlaGrq4uhg4diqKiIixbtgzjxo3DkSNHcPjw4dcWzEmTJmH58uUYM2YMIiIi0K1bN9y5cwdJSUmYP38+unXrVud+Len6/vXXX3H//n28++67UFFRQXJyMqKjoxEUFNTs52xLNOqbEELayPTp07FhwwZs2rQJVlZWcHFxwebNm2FmZtbi537//ffRu3dvDBkyBBMmTMDo0aMlJleJjIxESEgIYmJiYGFhAXd3dxw6dOi1rx0bG4vOnTtjwIAB8PDwgJubG+zt7SW2iYiIQG5uLnr16sV1T1tYWGDNmjWIj4+HjY0Nzp8/36jCp6amhrS0NHTv3h2enp6wsLCAr68vysvL22yaZ0VFRcTHx8PZ2Rm2trZYt24dYmNjERYW1iav11I01zchfKC5vutEc303ztSpU/H06VPs37+f7yikATTXNyGEEPIGoEJNCCGECBgNJiOEEBnz6uQnpGNr1hH18ePHWzsHIYQQQurQrELt7u6OXr164ZtvvkF+fn5rZyKEEELI/9esQn337l34+/tj37596NmzJ9zc3LB3797XzlxDCCGN8YZdjEI6qNb6HjerUOvq6iIgIABXr17FH3/8gT59+uDLL7+EkZERZs+ejfT09FYJRwh5s9ROrUk/+klHUFZWBkByOtjmaPFgMnt7exgaGqJLly5YunQpEhMTsWbNGjg7OyMhIQFvv/12S1+CEPKGUFBQgJqaGoqKiqCoqAg5ObowhcgexhjKysrw4MEDaGtrS83t3lTNLtRVVVU4cOAAEhMTkZycDEdHR3z//ffw8vJCUVERgoODMX78eNy4caNFAQkhbw6RSISuXbvi9u3buHPnDt9xCGkRbW3tZi0F+qpmFepZs2Zh165dYIzhs88+w7Jly/DOO+9wj3fq1AnfffcdjIyMWhyQEPJmUVJSQu/evan7m8g0RUXFFh9J12pWob5x4wb+85//wNPTs96VRnR1dekyLkJIs8jJydEUooT8f806ARQWFobx48dLFenq6mqkpaUBeHGuqanLqxFCCCFEUrMK9bBhw/D48WOp9mfPnmHYsGEtDkUIIYSQF5pVqF9eGPxljx49QqdOnVocihBCCCEvNOkctaenJ4AXIzOnTp0q0fVdU1ODa9euYcCAAa2bkBBCCHmDNalQa2m9WEOXMQYNDQ2oqqpyjykpKeHdd9+Fn59f6yYkhBBC3mBNKtSbNm0CAJiamiIoKIi6uQkhhJA21uxR361VpOPj42FqagoVFRU4OTnh/PnzDW7/9OlTzJw5E127doWysjL69OmD3377rVWyEEIIIULT6CNqe3t7pKSkoHPnzrCzs6tzMFmty5cvN+o59+zZg8DAQCQkJMDJyQlxcXFwc3NDZmYm9PX1pbavrKzEBx98AH19fezbtw/Gxsa4c+cOtLW1G/s2CCGEEJnS6EI9ZswYbvDY2LFjW+XFY2Nj4efnBx8fHwBAQkICDh06hMTERCxcuFBq+8TERDx+/BhnzpzhJjk3NTVtlSyEEEKIEIkYT+vJVVZWQk1NDfv27ZMo/FOmTMHTp09x4MABqX1GjhwJHR0dqKmp4cCBA9DT08PEiROxYMGCeqdqq6ioQEVFBXe/uLgYJiYmePbsGTQ1NVv9fRHSKEu0GnjsWfvlIITwori4GFpaWo2qRbwtTfPw4UPU1NTAwMBAot3AwACFhYV17pOTk4N9+/ahpqYGv/32G0JCQrBixQp888039b5OTEwMtLS0uJuJiUmrvg9CCCGkLTW667tz584Nnpd+WV2zlrUGsVgMfX19/PDDD5CXl4eDgwPu3r2L5cuXIywsrM59Fi1ahMDAQO5+7RE1IYQQIgsaXajj4uJa9YV1dXUhLy+P+/fvS7Tfv3+/3mXBunbtKrUiiYWFBQoLC1FZWQklJSWpfZSVletdOIQQQggRukYX6ilTprTqCyspKcHBwQEpKSncOWqxWIyUlBT4+/vXuc/AgQOxc+dOiMVibkH5W7duoWvXrnUWaUIIIUTWNfocdXFxscTfDd0aKzAwEOvXr8eWLVuQkZGBL774AqWlpdwocG9vbyxatIjb/osvvsDjx48xZ84c3Lp1C4cOHUJ0dDRmzpzZ6NckhBBCZEmTzlEXFBRAX18f2tradZ6vrl2so6amplHPOWHCBBQVFSE0NBSFhYWwtbXFkSNHuAFmeXl53JEzAJiYmOD3339HQEAArK2tYWxsjDlz5mDBggWNfRuEEEKITGn05VknTpzAwIEDoaCggBMnTjS4rZDXoW7KkHhCWsJ04aF6H8tVmVj/jnR5FiEdXlNqUaOPqF8uvkIuxIQQQkhH0qRFOV725MkTbNy4ERkZGQAAS0tL+Pj4QEdHp9XCEUIIIW+6Zk14kpaWBlNTU6xevRpPnjzBkydPsHr1apiZmSEtLa21MxJCCCFvrGYdUc+cORMTJkzA2rVruWuaa2pq8OWXX2LmzJn4888/WzUkIYQQ8qZq1hF1VlYWvvrqK4mJR+Tl5REYGIisrKxWC0cIIYS86ZpVqO3t7blz0y/LyMiAjY1Ni0MRQggh5IVGd31fu3aN+3v27NmYM2cOsrKy8O677wIAzp07h/j4eCxdurT1UxJCCCFvqEZfRy0nJweRSITXbd6UCU/4QNdRk/ZC11ETQurTJtdR3759u8XBCCGEENI0jS7UPXr0aMschBBCCKlDsyc8AYAbN24gLy8PlZWVEu2jR49uUShCCCGEvNCsQp2Tk4OPPvoIf/75p8R569qFOoR8jpoQQgiRJc26PGvOnDkwMzPDgwcPoKamhr/++gtpaWlwdHREampqK0ckhBBC3lzNOqI+e/Ys/vvf/0JXVxdycnKQk5PDoEGDEBMTg9mzZ+PKlSutnZMQQgh5IzXriLqmpgYaGhoAAF1dXdy7dw/AiwFnmZmZrZeOEEIIecM164j6nXfeQXp6OszMzODk5IRly5ZBSUkJP/zwA3r27NnaGQkhhJA3VrMKdXBwMEpLSwEAERER+PDDDzF48GB06dIFe/bsadWAhBBCyJusWYXazc2N+9vc3Bw3b97E48eP0blzZ27kNyGEEEJarkXXUQNAfn4+AMDExKTFYQghhBAiqVmDyaqrqxESEgItLS2YmprC1NQUWlpaCA4ORlVVVWtnJIQQQt5YzTqinjVrFpKSkrBs2TI4OzsDeHHJ1pIlS/Do0SOsXbu2VUMSQgghb6pmFeqdO3di9+7dGDFiBNdmbW0NExMTeHl5UaEmhBBCWkmzur6VlZVhamoq1W5mZgYlJaWWZiKEEELI/9esQu3v74/IyEhUVFRwbRUVFYiKioK/v3+rhSOEEELedI3u+vb09JS4f+zYMXTr1g02NjYAgPT0dFRWVuL9999v3YSEEELIG6zRhVpLS0vi/scffyxxny7PIoQQQlpfowv1pk2b2jIHIYQQQurQoglPioqKuEU43nrrLejp6bVKKEIIIYS80KzBZKWlpZg2bRq6du2KIUOGYMiQITAyMoKvry/KyspaOyMhhBDyxmpWoQ4MDMSJEyfw66+/4unTp3j69CkOHDiAEydO4Kuvvmry88XHx8PU1BQqKipwcnLC+fPnG7Xf7t27IRKJMHbs2Ca/JiGEECILmlWof/rpJ2zcuBEjRoyApqYmNDU1MXLkSKxfvx779u1r0nPt2bMHgYGBCAsLw+XLl2FjYwM3Nzc8ePCgwf1yc3MRFBSEwYMHN+ctEEIIITKhWYW6rKwMBgYGUu36+vpN7vqOjY2Fn58ffHx8YGlpiYSEBKipqSExMbHefWpqajBp0iSEh4fT+teEEEI6tGYVamdnZ4SFhaG8vJxre/78OcLDw7m5vxujsrISly5dgqur6/8CycnB1dUVZ8+erXe/iIgI6Ovrw9fX97WvUVFRgeLiYokbIYQQIiuaNeo7Li4O7u7uUhOeqKio4Pfff2/08zx8+BA1NTVSR+cGBga4efNmnfucOnUKGzduxNWrVxv1GjExMQgPD290JkIIIURImlWorays8Pfff2PHjh1cQfXy8sKkSZOgqqraqgFf9u+//+Kzzz7D+vXroaur26h9Fi1ahMDAQO5+cXExTc5CCCFEZjS5UFdVVaFv3744ePAg/Pz8WvTiurq6kJeXx/379yXa79+/D0NDQ6nts7OzkZubCw8PD65NLBYDABQUFJCZmYlevXpJ7KOsrAxlZeUW5SSEEEL40uRz1IqKihLnpltCSUkJDg4OSElJ4drEYjFSUlLqPNfdt29f/Pnnn7h69Sp3Gz16NIYNG4arV6/SkTIhhJAOp1ld3zNnzsS3336LDRs2QEGhRZObITAwEFOmTIGjoyP69++PuLg4lJaWwsfHBwDg7e0NY2NjxMTEQEVFBe+8847E/tra2gAg1U4IIYR0BM2qshcuXEBKSgqOHj0KKysrdOrUSeLxpKSkRj/XhAkTUFRUhNDQUBQWFsLW1hZHjhzhBpjl5eVBTq5Zg9MJIYQQmdesQq2trS21elZL+Pv717uOdWpqaoP7bt68udVyEEIIIULTpEItFouxfPly3Lp1C5WVlXjvvfewZMmSNh3pTQghhLzJmtSnHBUVhcWLF0NdXR3GxsZYvXo1Zs6c2VbZCCGEkDdek46ot27dijVr1uDzzz8HABw7dgyjRo3Chg0b6DwyIYR0cKYLD9XZnrt0VDsnebM0qbrm5eVh5MiR3H1XV1eIRCLcu3ev1YMRQgghpImFurq6GioqKhJtioqKqKqqatVQhBBCCHmhSV3fjDFMnTpVYqav8vJyzJgxQ+ISraZcnkUIIYSQ+jWpUE+ZMkWqbfLkya0WhhBCCCGSmlSoN23a1FY5CCGEEFIHGqpNCCGECBgVakIIIUTAqFATQgghAkaFmhBCCBEwKtSEEEKIgFGhJoQQQgSMCjUhhBAiYFSoCSGEEAGjQk0IIYQIGBVqQgghRMCoUBNCCCECRoWaEEIIETAq1IQQQoiAUaEmhBBCBIwKNSGEECJgVKgJIYQQAaNCTQghhAiYAt8BCCGSrLZY1fvYn1P+bMckhBAhoCNqQgghRMCoUBNCCCECJohCHR8fD1NTU6ioqMDJyQnnz5+vd9v169dj8ODB6Ny5Mzp37gxXV9cGtyeEEEJkGe/nqPfs2YPAwEAkJCTAyckJcXFxcHNzQ2ZmJvT19aW2T01NhZeXFwYMGAAVFRV8++23GD58OP766y8YGxvz8A4IIYTUh8ZctBzvR9SxsbHw8/ODj48PLC0tkZCQADU1NSQmJta5/Y4dO/Dll1/C1tYWffv2xYYNGyAWi5GSktLOyQkhhJC2x2uhrqysxKVLl+Dq6sq1ycnJwdXVFWfPnm3Uc5SVlaGqqgo6OjptFZMQQgjhDa9d3w8fPkRNTQ0MDAwk2g0MDHDz5s1GPceCBQtgZGQkUexfVlFRgYqKCu5+cXFx8wMTQggh7Yz3ru+WWLp0KXbv3o2ff/4ZKioqdW4TExMDLS0t7mZiYtLOKQkhhJDm47VQ6+rqQl5eHvfv35dov3//PgwNDRvc97vvvsPSpUtx9OhRWFtb17vdokWL8OzZM+6Wn5/fKtkJIYSQ9sBroVZSUoKDg4PEQLDagWHOzs717rds2TJERkbiyJEjcHR0bPA1lJWVoampKXEjhBBCZAXvl2cFBgZiypQpcHR0RP/+/REXF4fS0lL4+PgAALy9vWFsbIyYmBgAwLfffovQ0FDs3LkTpqamKCwsBACoq6tDXV2dt/dBCCGEtAXeC/WECRNQVFSE0NBQFBYWwtbWFkeOHOEGmOXl5UFO7n8H/mvXrkVlZSXGjRsn8TxhYWFYsmRJe0YnhBBC2hzvhRoA/P394e/vX+djqampEvdzc3PbPhAhhBAiEDI96psQQgjp6KhQE0IIIQJGhZoQQggRMEGco34T0UT1hBBCGoOOqAkhhBABo0JNCCGECBgVakIIIUTAqFATQgghAkaFmhBCCBEwKtSEEEKIgFGhJoQQQgSMCjUhhBAiYFSoCSGEEAGjQk0IIYQIGBVqQgghRMCoUBNCCCECRotyEEJajBaZIR2J0L7PdERNCCGECBgVakIIIUTAqOubNJrQuoMIIeRNQEfUhBBCiIBRoSaEEEIEjLq+W8h04aF6H8tdOqodkxBCCOmI6IiaEEIIETAq1IQQQoiAUdc36dBopDqpjyx+N2QxM2k5OqImhBBCBIwKNSGEECJgVKgJIYQQARNEoY6Pj4epqSlUVFTg5OSE8+fPN7j9jz/+iL59+0JFRQVWVlb47bff2ikpIYQQ0r54L9R79uxBYGAgwsLCcPnyZdjY2MDNzQ0PHjyoc/szZ87Ay8sLvr6+uHLlCsaOHYuxY8fi+vXr7ZycEEIIaXu8F+rY2Fj4+fnBx8cHlpaWSEhIgJqaGhITE+vcftWqVXB3d8e8efNgYWGByMhI2Nvb4/vvv2/n5IQQQkjb4/XyrMrKSly6dAmLFi3i2uTk5ODq6oqzZ8/Wuc/Zs2cRGBgo0ebm5ob9+/e3ZVRCCCH1WaJV/2Nm3dsvRwfFa6F++PAhampqYGBgINFuYGCAmzdv1rlPYWFhndsXFhbWuX1FRQUqKiq4+8+ePQMAFBcXtyQ6R1xRVu9jDb1GzfOaZu3XGt4J+73ex66Hu9X7GJ+Zm4vPzA1+N0Ss3sf4/pzr+37Qd4N/fGeu7ztN3+emq30exur/7DiMR3fv3mUA2JkzZyTa582bx/r371/nPoqKimznzp0SbfHx8UxfX7/O7cPCwhgAutGNbnSjG90Ed8vPz39treT1iFpXVxfy8vK4f/++RPv9+/dhaGhY5z6GhoZN2n7RokUSXeVisRiPHz9Gly5dIBKJWvgOJBUXF8PExAT5+fnQ1NRs1eduK5S5fVDm9kGZ2wdlbjnGGP79918YGRm9dlteC7WSkhIcHByQkpKCsWPHAnhRSFNSUuDv71/nPs7OzkhJScHcuXO5tuTkZDg7O9e5vbKyMpSVlSXatLW1WyN+vTQ1NQXxRWgKytw+KHP7oMztgzK3jJaWVqO2432u78DAQEyZMgWOjo7o378/4uLiUFpaCh8fHwCAt7c3jI2NERMTAwCYM2cOXFxcsGLFCowaNQq7d+/GxYsX8cMPP/D5NgghhJA2wXuhnjBhAoqKihAaGorCwkLY2triyJEj3ICxvLw8yMn97yqyAQMGYOfOnQgODsbixYvRu3dv7N+/H++88w5fb4EQQghpM7wXagDw9/evt6s7NTVVqm38+PEYP358G6dqOmVlZYSFhUl1tQsZZW4flLl9UOb2QZnbl4ixxowNJ4QQQggfeJ+ZjBBCCCH1o0JNCCGECBgVakIIIUTAqFATQgghAkaFupmqq6uxdetWqVnSCCGEkNZEo75bQE1NDRkZGejRowffURptypQp8PX1xZAhQ/iO0iQ9e/bEhQsX0KVLF4n2p0+fwt7eHjk5OTwl+59ffvml0duOHj26DZO82WpqavDnn3+iR48e6Ny5M99xZFZTFp8Qykxfr0pLS2vwcVn5d1AQ11HLqv79++Pq1asyVaifPXsGV1dX9OjRAz4+PpgyZQqMjY35jvVaubm5qKmRXtGmoqICd+/e5SGRtNppcGuJRCKJlXFenlu+rvciBFu2bIGuri5GjRoFAJg/fz5++OEHWFpaYteuXYL8rs+dOxdWVlbw9fVFTU0NXFxccObMGaipqeHgwYMYOnQo3xFlkra2dqPXQxDq97mu/+9l4b/DV1GhboEvv/wSgYGByM/Ph4ODAzp16iTxuLW1NU/J6rd//34UFRVh27Zt2LJlC8LCwuDq6gpfX1+MGTMGioqKfEeU8PJR6u+//y4xN25NTQ1SUlJgamrKQzJpYrGY+/vYsWNYsGABoqOjuXnoz549i+DgYERHR/MV8bWio6Oxdu1aAC/yxsfHY+XKlTh48CACAgKQlJTEc0Jp+/btw+TJkwEAv/76K27fvo2bN29i27Zt+Prrr3H69GmeE9Zt37592Lt3L/Ly8lBZWSnx2OXLl3lK9T/Hjx/n/s7NzcXChQsxdepUie/zli1buOmdhejJkycS96uqqnDlyhWEhIQgKiqKp1TN8Nr1tUi9RCKR1E1OTo77X1lw6dIl5u/vz1RUVJiuri6bO3cuu3XrFt+xOHV9xrU3JSUl1qdPH/brr7/yHVPK22+/zU6ePCnVnpaWxvr27ctDosZRVVVld+7cYYwxNn/+fPbZZ58xxhi7fv0609XV5TNavZSVlbmlAv38/NicOXMYY4zl5OQwDQ0NHpPVb9WqVUxdXZ35+/szJSUl9vnnnzNXV1empaXFFi9ezHc8Ke+9957U8sKMMbZjxw7m4uLS/oFaKDU1ldnb2/Mdo9FoMFkL3L59W+qWk5PD/a/QFRQUIDk5GcnJyZCXl8fIkSPx559/wtLSEitXruQ7HoAXR6lisRg9evRAUVERd18sFqOiogKZmZn48MMP+Y4pJTs7u85V2rS0tJCbm9vueRpLXV0djx49AgAcPXoUH3zwAQBARUUFz58/5zNavQwMDHDjxg3U1NTgyJEjXOaysjLIy8vznK5ua9aswQ8//ID//Oc/UFJSwvz585GcnIzZs2fj2bNnfMeTcvbsWTg6Okq1Ozo64vz58zwkahkDAwNkZmbyHaPx+P6lQNpXZWUl27dvHxs1ahRTVFRkDg4ObO3atezZs2fcNklJSUxbW5vHlJIqKyvZe++9J6gj/dcZPHgw++CDD1hhYSHXVlhYyIYPH86GDBnCY7KGTZw4kdnb2zNfX1+mpqbGHj58yBhj7MCBA+ztt9/mOV3dwsLCmJaWFuvbty/r3r07Ky8vZ4wxtnHjRvbuu+/ynK5uqqqqLDc3lzHGmJ6eHrt69SpjjLFbt24xHR0dPqPVqU+fPmzevHlS7fPmzWN9+vThIVHjpKenS9yuXr3KDh8+zFxcXNjAgQP5jtdodI66hbZt24aEhATcvn0bZ8+eRY8ePRAXFwczMzOMGTOG73hSunbtCrFYDC8vL5w/fx62trZS2wwbNqzN1+xuCkVFRVy7do3vGE2yceNGeHp6onv37jAxMQEA5Ofnc6u9CVV8fDyCg4ORn5+Pn376iRtlf+nSJXh5efGcrm5LlizBO++8g/z8fIwfP55bdEFeXh4LFy7kOV3dDA0N8fjxY/To0QPdu3fHuXPnYGNjg9u3b0sMQBSKlStX4uOPP8bhw4fh5OQEADh//jz+/vtv/PTTTzynq5+tra3UoE4AePfdd5GYmMhTqqajy7NaYO3atQgNDcXcuXMRFRWF69evo2fPnti8eTO2bNkiMRhDKLZt24bx48dDRUWF7yhNEhAQAGVlZSxdupTvKI3GGENycjJu3rwJALCwsICrq2ujR9KSpisvL5eJ7/b06dNhYmKCsLAwxMfHY968eRg4cCAuXrwIT09PbNy4ke+IUv755x+sXbsWGRkZAF58n2fMmMH9EBWiO3fuSNyXk5ODnp6eTHxHXkaFugUsLS0RHR2NsWPHQkNDA+np6ejZsyeuX7+OoUOH4uHDh3xHlFBVVQVVVVVcvXpV5tbvnjVrFrZu3YrevXvXOcI+NjaWp2TSZPlzBoCTJ09i3bp1yMnJwY8//ghjY2Ns27YNZmZmGDRoEN/xpNTU1CA6OhoJCQm4f/8+bt26hZ49eyIkJASmpqbw9fXlO6KU2nEWCgovOjV3796NM2fOoHfv3vj888+hpKTEc8L/qaqqgru7OxISEtC7d2++47yRaDBZC9y+fRt2dnZS7crKyigtLeUhUcMUFRXRvXt3mbl28GXXr1+Hvb09NDQ0cOvWLVy5coW7Xb16le94EmT5c/7pp5/g5uYGVVVVXL58GRUVFQBeXH8v1MvKoqKisHnzZixbtkyiwL3zzjvYsGEDj8nqJycnxxVpAPj000+xevVqzJo1S1BFGpDNU08vO3HiBDw8PGBubg5zc3OMHj0aJ0+e5DtW0/B4flzmWVhYsP379zPGGFNXV2fZ2dmMMcZWr17N7Ozs+IxWrw0bNrCRI0eyR48e8R2lQ5PVz9nW1pZt2bKFMSb5nb58+TIzMDDgM1q9evXqxY4dO8YYk8yckZEhqEGRLzMzM2NTp07lBr7VKioqYmZmZjylqt/cuXPZggUL+I7RZNu2bWMKCgrsk08+YatWrWKrVq1in3zyCVNUVGQ7duzgO16j0WCyFggMDMTMmTNRXl4OxhjOnz+PXbt2ISYmRrC/5L///ntkZWXByMgIPXr0kOpCFsJEC6/zzz//AAC6devGc5L6yernnJmZWee0ilpaWnj69Gn7B2qEu3fvwtzcXKpdLBajqqqKh0Svl5ubCwUFBQwePBi//PILDA0NAbzoxn/1vKoQVFdXIzExEceOHRP8qaeXRUVFYdmyZQgICODaZs+ejdjYWERGRmLixIk8pms8KtQtMH36dKiqqiI4OBhlZWWYOHEijIyMsGrVKnz66ad8x6vTq9NcygqxWIxvvvkGK1asQElJCQBAQ0MDX331Fb7++mvIyQnrLI6sfs6GhobIysqSmu3t1KlT6NmzJz+hXsPS0hInT56Umt503759dZ6aEgKRSIQjR44gKCgIDg4O2L9/P/r168d3rHrVnnoCgFu3bkk8JuTBkTk5OfDw8JBqHz16NBYvXsxDombi+5C+oygtLWX379/nO0aHtXDhQqanp8fWrFnDXRMZHx/P9PT0BDmTk6yKjo5mlpaW7Ny5c0xDQ4OdPHmSbd++nenp6bHVq1fzHa9O+/fvZ1paWmzp0qVMTU2NLV++nE2fPp0pKSmxo0eP8h2vTiKRiPv3YuHChUxVVZVt27aNFRYWysyshrKgV69eLCEhQap97dq1zNzcnIdEzUOFugXKyspYaWkpdz83N5etXLmS/f777zymer0nT56w9evXs4ULF3LnUC9dusT++ecfnpPVr2vXruzAgQNS7fv372dGRkY8JOqYxGIx++abb1inTp24qVpVVFRYcHAw39EalJaWxlxdXZmenh5TVVVlAwcOFPR/h3JychI/7Ldt28ZUVFSYj48PFepWtGbNGqakpMRmzJjBtm7dyrZu3co+//xzpqysXGcBFyq6PKsFhg8fDk9PT8yYMQNPnz7FW2+9BSUlJTx8+BCxsbH44osv+I4o5dq1a3B1deWmsszMzETPnj0RHByMvLw8bN26le+IdVJRUcG1a9fQp08fifbMzEzY2toKbnrLmpoarFy5st5FFx4/fsxTssaprKxEVlYWSkpKYGlpCXV1db4jdShycnIoLCyEvr4+13b27Fl89NFHKCoqEuQVAxcvXqz3+yzExVpq/fzzz1ixYoXE9d/z5s0T5IRU9eL7l4Is69KlC7t+/TpjjLH169cza2trVlNTw/bu3SvYhRfef/99birAl0fInj59mvXo0YPHZA3r378/mzVrllS7v78/c3Jy4iFRw0JCQljXrl3Zd999x1RUVFhkZCTz9fVlXbp0YatWreI7Xofi6+vLjh8/zneMVlFYWMhSU1P5jiFl165dTFFRkX344YdMSUmJffjhh6xPnz5MS0uLTZ06le949fL29mYnTpzgO0aLUaFugZdXGho/fjxbsmQJY4yxvLw8pqqqyme0emlqarKsrCzGmGShzs3NZcrKynxGa1Bqairr1KkTs7CwYNOmTWPTpk1jFhYWTF1dnaWlpfEdT0rPnj3ZwYMHGWMvPufaz3zVqlXMy8uLz2gNKikpYcHBwczZ2Zn16tWLmZmZSdyEaPTo0UxZWZl169aNBQUFsStXrvAd6bXCw8NZSkqKVHtJSQkLDw/nIVHDrKys2Pfff88Y+9+/G2KxmPn5+bHQ0FCe09VvzJgxTFFRkZmbm7OoqCh29+5dviM1CxXqFrCysmKrVq1ieXl5TFNTk505c4YxxtjFixcFe82pnp4eu3z5MmNMslAfPXqUdevWjc9or3X37l22ePFi5unpyTw9PdnXX38t2P/w1NTUuB9xhoaG7NKlS4wxxrKzs5mmpiaf0Rr06aefsq5du7L58+ezlStXsri4OImbUD1+/JitW7eOubi4MDk5OWZpacmioqLY7du3+Y5Wp9plWlesWCHRLtTBZGpqatxnqaOjw65du8YYY+zGjRvM0NCQx2Sv9+DBA7ZixQpmbW3NFBQUmLu7O9u7dy+rrKzkO1qjUaFugR9//JEpKioyOTk55urqyrVHR0czd3d3HpPVz9fXl40dO5ZVVlYydXV1lpOTw+7cucPs7Oy4dXyF4qOPPuJW9dqyZYvU5BBC1qdPH3bu3DnGGGMDBw5kMTExjDHGdu/ezfT09PiM1iAtLS126tQpvmO0SH5+Plu2bBnr27cvk5eX5ztOnUQiEdu9ezfr0qULmzp1KquoqGCMCbdQGxsbc8XZysqKW5v6zJkzgv7h+apLly4xf39/pqKiwnR1ddncuXNlYlU+KtQtVFBQwC5fvsxqamq4tj/++INlZGTwmKp+T58+Za6urkxbW5vJy8szExMTpqioyIYMGcJKSkr4jidBUVGR3bt3jzEmPUpW6BYsWMCioqIYYy+Ks4KCAjM3N2dKSkqCnuHJ1NSU3bhxg+8YzVZZWcl+/vln9vHHHzMVFRXBXhFQe3lWVlYWs7CwYM7Ozuz+/fuCLdReXl7c0X9ERATT09Nj06dPZz169GAfffQRz+ka5969e2zp0qXsrbfeYp06dWLe3t7s/fffZwoKCiw2NpbveA2iUd+tRBZmy3rZqVOncO3aNZSUlMDe3h6urq58R5JibW0Ne3t7DBs2DD4+Pli9ejU0NTXr3Nbb27ud0zXNuXPnuEUX6pqAQSi2b9+OAwcOYMuWLVBTU+M7TqMdP34cO3fuxE8//QSxWAxPT09MmjQJ7733niAn5JCXl0dBQQH09fVRXFyMTz75BH/99RcSEhIwevRowY36fvz4McrLy2FkZASxWIxly5Zx3+fg4GB07tyZ74h1qqqqwi+//IJNmzbh6NGjsLa2xvTp0zFx4kTu35Kff/4Z06ZNw5MnT3hOWz8q1C0ga7NlAS/WRBbysnQvO336NL766itkZ2fj8ePH0NDQqPMfXZFIJPjLnYTMzs5O4nPNysoCYwympqZQVFSU2FaIU58aGxvj8ePHcHd3x6RJk+Dh4cGtSS1Ur16eJRaLMXfuXKxduxZisVhwhVpW6erqQiwWw8vLC35+frC1tZXa5unTp7Czs8Pt27fbP2Aj0RSiLfD1119j48aNWLp0KQYOHAjgxZHqkiVLUF5ejqioKJ4TSjM1NcWgQYMwefJkjBs3TrC/hAFg4MCBOHfuHIAX/7DdunVL4rpTIevevTuGDh0KFxcXDB06FL169eI7Ur1kdbrTWkuWLMH48eOhra3Nd5RG27RpE7S0tLj7cnJyWL16Nezs7JCWlsZjsrp5e3tj2LBhGDJkiKC/y69auXIlxo8f3+D609ra2oIu0gAdUbeIkZER11X1sgMHDuDLL7/E3bt3eUpWvytXrmDnzp3YvXs3ioqK4O7ujsmTJwvyKMTT0xObN2+GpqYmtmzZgk8++QSqqqp8x2qU7du3Iy0tDampqcjKyoKxsTFcXFy4wk3r+rYNWTsFJSumT5+OtLQ0ie9y7Q9R+i63PSrULSBrs2W9jDGG1NRUqfN6iYmJfEfjKCkp4c6dO+jatavEOT1ZU1BQgBMnTuDgwYPYs2ePoLs2L1y4ALFYDCcnJ4n2P/74A/Ly8nB0dOQpWf1k5RTU6tWr8X//939QUVHB6tWr691OJBJh1qxZ7Zis8e7evYu0tDScOHECJ06cwK1bt9C1a1fuBxJpG1SoW8DJyQlOTk5S/9HNmjULFy5c4Lpthe7y5cvw9fXFtWvXBFVAZH0wWVlZGU6dOoXU1FQcP34cV65cgYWFBYYOHYqVK1fyHa9O/fv3x/z58zFu3DiJ9qSkJHz77bf4448/eEpWv0WLFmHjxo0IDw+XOgXl5+cnmFNQZmZmuHjxIrp06QIzM7N6txOJRMjJyWnHZI1X+50+fvw4UlNTcfnyZVhaWuLKlSt8R+vQqFC3wIkTJzBq1Ch0794dzs7OAF7M15ufn4/ffvsNgwcP5jlh/f755x/s3LkTO3fuxPXr1+Hs7IxJkyZhxowZfEfjnDlzBoGBgTI5mGzAgAEShdnFxQVDhgwR9JgAAFBXV8e1a9eklrS8ffs2rK2t8e+///KUrH6yeArqZbX/BAtxdHqtxYsXIzU1lftO13Z9y8J3uiOgQt1C9+7dQ3x8PG7evAngxYTvX375JYyMjHhOVrd169Zh586dOHXqFCwsLDBp0iRMnDhRai1foalrEQMh09HRgZycHIYPH46hQ4di6NChUqdIhKhLly44ePAg98Oz1pkzZzBq1ChBXsIiq6egNm7ciJUrV+Lvv/8GAPTu3Rtz587F9OnTeU4mTU5ODnp6eggICICnp6dMfJc7EirUbxgTExN4eXlh0qRJsLGx4TtOo925cwd5eXlYt24dcnJy8OOPP8LY2Bjbtm2DmZkZBg0axHdECYwx/Pnnn0hNTcWJEyeQlpYGJSUluLi4YNiwYfDz8+M7Yp28vLxQUFCAAwcOcKOSnz59irFjx0JfXx979+7lOaE0WTwFFRoaitjYWMyaNUuiN+77779HQEAAIiIieE4oKT09HSdOnEBqaipOnjzJfZdl6UeoLKNC3UTXrl1r9LbW1tZtmKR5GGM4deqUzBS8Wj/99BM+++wzTJo0Cdu2bcONGzfQs2dPfP/99/jtt9/w22+/8R2xXowxXLp0Cd9//z127Ngh6MFkd+/exZAhQ/Do0SPY2dkBAK5evQoDAwMkJycL8hr8+k5B5eXl4fDhw4I8BaWnp4fVq1fDy8tLon3Xrl2YNWsWHj58yFOyxklPT8fKlSsF/33uKOg66iaytbWFSCTC637fiEQiQX55k5KSuIJ3+fJlVFRUAACePXuG6OhowRa8b775BgkJCfD29sbu3bu59oEDB+Kbb77hMVndLl++jNTUVKSmpuLUqVP4999/YWVlhVmzZsHFxYXvePUyNjbGtWvXsGPHDqSnp0NVVRU+Pj7w8vKSmvxEKFxcXJCZmYm1a9dyaw57enoK+hRUVVVVnSPoHRwcUF1dzUOihjHGcOXKFYnvdHFxMaytrQX9fe4o6Ii6ie7cudPobYV43tfOzg4BAQHw9vaGhoYG0tPT0bNnT1y5cgUjRoxAYWEh3xHrpKamhhs3bsDU1FQid05ODiwtLVFeXs53RAkKCgqws7Pjrp0eMmSIxAQXpHWVl5fj2rVrePDgAcRiscRjrw4yE4JZs2ZBUVERsbGxEu1BQUF4/vw54uPjeUpWt86dO6OkpAQ2NjZcl/fgwYNlapIZWUZH1E30cvGNiYmBgYEBpk2bJrFNYmIiioqKsGDBgvaO91qZmZkYMmSIVLuWlhaePn3a/oEaydDQEFlZWTA1NZVoP3XqlNQIZb7V1NQgKSkJgwcPlskRsX///TeOHz9eZ9ELDQ3lKVX9jhw5Am9vbzx69Eiqp0uoPVvAi8FkR48exbvvvgvgxbXqeXl58Pb2RmBgILfdq8WcD9u3b8fgwYPrvTyStC0q1C1QO4L6VW+//TY+/fRTQRZqWSp4L/Pz88OcOXOQmJgIkUiEe/fu4ezZswgKCkJISAjf8STIy8vjk08+QUZGhswV6vXr1+OLL76Arq4uDA0NJS4ZEolEgizUs2bNwvjx4xEaGgoDAwO+4zTK9evXYW9vDwDIzs4G8GJeal1dXVy/fp3bTiiXbI0aNYr7m2Z/40G7rNHVQSkrK7OcnByp9uzsbKasrMxDoteLjo5mlpaW7Ny5c0xDQ4OdPHmSbd++nenp6bHVq1fzHa9eYrGYffPNN6xTp05MJBIxkUjEVFRUWHBwMN/R6uTg4MCOHTvGd4wm6969O1u6dCnfMZpEQ0ODZWVl8R2jQ6upqWHh4eFMU1OTycnJMTk5OaalpcUiIiIklvglbYMKdQuYm5uzbdu2SbVv3bqVmZmZ8ZDo9WSt4L2qoqKC/fXXX+yPP/5g//77L99x6nX48GFma2vLfv31V3bv3j327NkziZtQaWhosOzsbL5jNImPjw/bsGED3zE6tIULFzI9PT22Zs0alp6eztLT01l8fDzT09Njixcv5jteh0eDyVpg2bJlWLZsGZYvX4733nsPAJCSkoL58+fjq6++wqJFi3hOWL/KykpkZWWhpKQElpaWUFdX5ztSh/Ly/NIvd18yxgR93tTX1xf9+vUT1Ax1r1NWVobx48dDT08PVlZWUqPTZ8+ezVOyjkPWZ3+TdXSOugXmzZuHR48e4csvv0RlZSWAF7MkLViwQNBFGnix4IWlpSXfMTqs48eP8x2hWczNzRESEoJz587JTNHbtWsXjh49ChUVFaSmpkqdVxdiZlnz+PFj9O3bV6q9b9++gpu+tyOiI+pWUFJSgoyMDKiqqqJ3796CWy6SkMaSxcUiDA0NMXv2bCxcuFAwK2V1NLI4+1tHQoWakDby9OlTbNy4kZuE4+2338a0adPoeupWpqOjgwsXLqBXr158R+mwZHkBoo6ACjUhbeDixYtwc3ODqqoq+vfvD+DFWs/Pnz/H0aNHuUtzhCAwMBCRkZHo1KmTxPW7rxKJRFixYkU7JmucgIAA6OnpYfHixXxH6bDy8vKgoKBQ5wJE1dXV6N69O88JOzYq1IS0gcGDB8Pc3Bzr16+HgsKLoSDV1dWYPn06cnJykJaWxnPC/xk2bBh+/vlnaGtrY9iwYfVuJxKJ8N///rcdkzXO7NmzsXXrVtjY2MDa2lrqvLoQJgyRdfLy8igoKJBave7Ro0fQ19cX7ODIjoIKNSFtQFVVFVeuXJEagHPjxg04OjqirKyMp2Qdjyz+uJA19S0ze+fOHVhaWqK0tJSnZG8GGvVNSBvQ1NREXl6eVKHOz8+HhoYGT6k6JlkdYS8Lak+F1M5Kp6amxj1WU1ODP/74A7a2tjyle3NQoSakDUyYMAG+vr747rvvMGDAAADA6dOnMW/ePKmlDQkRqitXrgD43/rqSkpK3GNKSkqwsbFBUFAQX/HeGNT1TUgruXbtGt555x3IycmhsrIS8+bNQ0JCArdsoaKiIr744gssXbqULuEjMsXHxwerVq2iRTl4QoWakFby8oCbnj174sKFC1BVVeUWXejVq5dE1yEhhDQGdX0T0kq0tbVx+/Zt6OvrIzc3F2KxGGpqarCysuI7GiFEhlGhJqSVfPzxx3BxcUHXrl0hEong6OgIeXn5OrcV4gxfhBBhokJNSCv54Ycf4OnpiaysLMyePRt+fn40wpsQ0mJ0jpqQNuDj44PVq1dToSaEtBgVakIIIUTAaKkZQgghRMCoUBNCCCECRoWaEEIIETAq1IQQQoiAUaEmhBBCBIwKNSGEECJgVKgJIYQQAaNCTQghhAjY/wM4jaWa+Um4+AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting\n",
    "x = torch.arange(len(vocab))\n",
    "bar_width = 0.15\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 3))\n",
    "for i, T in enumerate(temperatures):\n",
    "    rects = ax.bar(x + i * bar_width, scaled_probas[i], bar_width, label=f'Temperature = {T}')\n",
    "\n",
    "ax.set_ylabel('Probability')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(vocab.keys(), rotation=90)\n",
    "ax.legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"temperature-plot.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "90c44c48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 x closer\n",
      "0 x every\n",
      "0 x effort\n",
      "985 x forward\n",
      "0 x inches\n",
      "0 x moves\n",
      "0 x pizza\n",
      "15 x toward\n",
      "0 x you\n"
     ]
    }
   ],
   "source": [
    "print_sampled_tokens(scaled_probas[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "f3d12846",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "73 x closer\n",
      "0 x every\n",
      "0 x effort\n",
      "582 x forward\n",
      "2 x inches\n",
      "0 x moves\n",
      "0 x pizza\n",
      "343 x toward\n",
      "0 x you\n"
     ]
    }
   ],
   "source": [
    "print_sampled_tokens(probas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "24632654",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "165 x closer\n",
      "75 x every\n",
      "42 x effort\n",
      "239 x forward\n",
      "71 x inches\n",
      "46 x moves\n",
      "32 x pizza\n",
      "227 x toward\n",
      "103 x you\n"
     ]
    }
   ],
   "source": [
    "print_sampled_tokens(scaled_probas[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3203e57",
   "metadata": {},
   "source": [
    "### Top-K采样"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bef3485",
   "metadata": {},
   "source": [
    "- 为了能够使用较高的温度值来增加输出多样性，同时降低生成无意义语句的概率，我们可以将采样范围限制在​​概率最高的前k个词元​​内；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd0995c9",
   "metadata": {},
   "source": [
    "![](./images/topk.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "493b0d00",
   "metadata": {},
   "outputs": [],
   "source": [
    "next_token_logits = torch.tensor(\n",
    "    [4.51, 0.89, -1.90, 6.75, 1.63, -1.62, -1.89, 6.28, 1.79]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "4bb86ec5",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_k = 3\n",
    "\n",
    "top_logits, top_pos = torch.topk(next_token_logits, top_k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "657c7437",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_logits = torch.where(\n",
    "    condition=next_token_logits<top_logits[-1],\n",
    "    input=torch.tensor(float(\"-inf\")),\n",
    "    other=next_token_logits,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "ed454c9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0615, 0.0000, 0.0000, 0.5775, 0.0000, 0.0000, 0.0000, 0.3610, 0.0000])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_probs = torch.softmax(new_logits, dim=0)\n",
    "\n",
    "top_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "d14448f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.)"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_probs.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3ad086c",
   "metadata": {},
   "source": [
    "### 修改文本生成函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "b5345dda",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate(model, idx, max_new_tokens, context_size, temperature=0.0, top_k=None, eos_id=None):\n",
    "    for _ in range(max_new_tokens):\n",
    "        idx_cond = idx[:, -context_size:]\n",
    "        with torch.no_grad():\n",
    "            logits = model(idx_cond)\n",
    "        logits = logits[:, -1, :]\n",
    "\n",
    "        if top_k is not None:\n",
    "            top_logits, _ = torch.topk(logits, top_k)\n",
    "            min_val = top_logits[:, -1]\n",
    "            logits = torch.where(logits < min_val, torch.tensor(float(\"-inf\")).to(logits.device), logits)\n",
    "\n",
    "        if temperature > 0.0:\n",
    "            logits = logits / temperature\n",
    "            probs = torch.softmax(logits, dim=-1)\n",
    "            idx_next = torch.multinomial(probs, num_samples=1)\n",
    "        else:\n",
    "            idx_next = torch.argmax(logits, dim=-1, keepdim=True)\n",
    "\n",
    "        if idx_next == eos_id:\n",
    "            break\n",
    "\n",
    "        idx = torch.cat((idx, idx_next), dim=1) \n",
    "\n",
    "    return idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "0a7cb4cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort moves you, I turned to what the house.\"\n",
      "\n",
      "\n",
      "I made him--\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"Every effort moves you\", tokenizer),\n",
    "    max_new_tokens=15,\n",
    "    context_size=GPT_CONFIG_124M[\"context_length\"],\n",
    "    top_k=25,\n",
    "    temperature=1.4\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bbfa045",
   "metadata": {},
   "source": [
    "## 加载和存储模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7cd2b1",
   "metadata": {},
   "source": [
    "![](./images/mental-model.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "9108ee72",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), \"model.pth\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "f101e9e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_model = GPTModel(GPT_CONFIG_124M)\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "loaded_model.load_state_dict(torch.load(\"model.pth\", map_location=device, weights_only=True))\n",
    "loaded_model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "061de9a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort moves you?\"\n",
      "\n",
      "The, pushed one of his pictures-- tone told me--\n"
     ]
    }
   ],
   "source": [
    "token_ids = generate(\n",
    "    model=loaded_model,\n",
    "    idx=text_to_token_ids(\"Every effort moves you\", tokenizer),\n",
    "    max_new_tokens=15,\n",
    "    context_size=GPT_CONFIG_124M[\"context_length\"],\n",
    "    top_k=25,\n",
    "    temperature=1.4\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e28a1f5",
   "metadata": {},
   "source": [
    "- 通常使用自适应优化器（如Adam或AdamW）而非标准SGD来训练大语言模型；\n",
    "- 这些自适应优化器会为每个模型权重存储额外的参数，因此若计划后续继续预训练，建议同步保存这些优化器状态；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "024800fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save({\n",
    "    \"model_state_dict\": model.state_dict(),\n",
    "    \"optimizer_state_dict\": optimizer.state_dict(),\n",
    "    }, \n",
    "    \"model_and_optimizer.pth\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "3a5bc0ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = torch.load(\"model_and_optimizer.pth\", weights_only=True)\n",
    "\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "model.load_state_dict(checkpoint[\"model_state_dict\"])\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=0.0005, weight_decay=0.1)\n",
    "optimizer.load_state_dict(checkpoint[\"optimizer_state_dict\"])\n",
    "model.train();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d43f83",
   "metadata": {},
   "source": [
    "## 加载OpenAI模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "b13969a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025-10-05 16:29:42.782942: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
      "E0000 00:00:1759652982.857393    8968 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "E0000 00:00:1759652982.878067    8968 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "W0000 00:00:1759652983.112817    8968 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
      "W0000 00:00:1759652983.112866    8968 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
      "W0000 00:00:1759652983.112868    8968 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
      "W0000 00:00:1759652983.112871    8968 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.\n",
      "2025-10-05 16:29:43.132438: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "from gpt_download import download_and_load_gpt2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "e3e76f6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "checkpoint: 100%|██████████| 77.0/77.0 [00:00<00:00, 67.0kiB/s]\n",
      "encoder.json: 100%|██████████| 1.04M/1.04M [00:18<00:00, 56.3kiB/s]\n",
      "hparams.json: 100%|██████████| 90.0/90.0 [00:00<00:00, 97.2kiB/s]\n",
      "model.ckpt.data-00000-of-00001: 100%|██████████| 498M/498M [2:14:04<00:00, 61.9kiB/s]  \n",
      "model.ckpt.index: 100%|██████████| 5.21k/5.21k [00:00<00:00, 6.32MiB/s]\n",
      "model.ckpt.meta: 100%|██████████| 471k/471k [00:02<00:00, 190kiB/s]  \n",
      "vocab.bpe: 100%|██████████| 456k/456k [00:02<00:00, 188kiB/s]  \n",
      "2025-10-05 18:44:46.773259: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.\n"
     ]
    }
   ],
   "source": [
    "settings, params = download_and_load_gpt2(model_size=\"124M\", models_dir=\"gpt2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77bc89c2",
   "metadata": {},
   "source": [
    "- 此外，\"355M\"、\"774M\"和\"1558M\"同样可作为支持的model_size参数\n",
    "- 这些不同规模模型之间的差异总结如下图所示："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f0aa53",
   "metadata": {},
   "source": [
    "![](./images/gpt-sizes.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90e58abd",
   "metadata": {},
   "source": [
    "- 上文我们已将124M参数规模的GPT-2模型权重加载至Python环境，但仍需将其载入我们自定义的GPTModel实例中；\n",
    "- 首先初始化一个新的GPTModel实例；\n",
    "- 需注意原始GPT模型在多头注意力模块中为查询、键、值矩阵的线性层初始化了偏置向量，此举并非必要亦不推荐；但为实现正确的权重加载，我们必须在实现中同样通过设置qkv_bias为True来启用这些偏置向量；\n",
    "- 我们同时采用原始GPT-2模型使用的1024词元上下文长度；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "d0f032c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define model configurations in a dictionary for compactness\n",
    "model_configs = {\n",
    "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
    "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
    "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
    "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
    "}\n",
    "\n",
    "# Copy the base configuration and update with specific model settings\n",
    "model_name = \"gpt2-small (124M)\"  # Example model name\n",
    "NEW_CONFIG = GPT_CONFIG_124M.copy()\n",
    "NEW_CONFIG.update(model_configs[model_name])\n",
    "NEW_CONFIG.update({\"context_length\": 1024, \"qkv_bias\": True})\n",
    "\n",
    "gpt = GPTModel(NEW_CONFIG)\n",
    "gpt.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "3dba0564",
   "metadata": {},
   "outputs": [],
   "source": [
    "def assign(left, right):\n",
    "    if left.shape != right.shape:\n",
    "        raise ValueError(f\"Shape mismatch. Left: {left.shape}, Right: {right.shape}\")\n",
    "    return torch.nn.Parameter(torch.tensor(right))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "44c38c8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def load_weights_into_gpt(gpt, params):\n",
    "    gpt.pos_emb.weight = assign(gpt.pos_emb.weight, params['wpe'])\n",
    "    gpt.tok_emb.weight = assign(gpt.tok_emb.weight, params['wte'])\n",
    "    \n",
    "    for b in range(len(params[\"blocks\"])):\n",
    "        q_w, k_w, v_w = np.split(\n",
    "            (params[\"blocks\"][b][\"attn\"][\"c_attn\"])[\"w\"], 3, axis=-1)\n",
    "        gpt.trf_blocks[b].att.W_query.weight = assign(\n",
    "            gpt.trf_blocks[b].att.W_query.weight, q_w.T)\n",
    "        gpt.trf_blocks[b].att.W_key.weight = assign(\n",
    "            gpt.trf_blocks[b].att.W_key.weight, k_w.T)\n",
    "        gpt.trf_blocks[b].att.W_value.weight = assign(\n",
    "            gpt.trf_blocks[b].att.W_value.weight, v_w.T)\n",
    "\n",
    "        q_b, k_b, v_b = np.split(\n",
    "            (params[\"blocks\"][b][\"attn\"][\"c_attn\"])[\"b\"], 3, axis=-1)\n",
    "        gpt.trf_blocks[b].att.W_query.bias = assign(\n",
    "            gpt.trf_blocks[b].att.W_query.bias, q_b)\n",
    "        gpt.trf_blocks[b].att.W_key.bias = assign(\n",
    "            gpt.trf_blocks[b].att.W_key.bias, k_b)\n",
    "        gpt.trf_blocks[b].att.W_value.bias = assign(\n",
    "            gpt.trf_blocks[b].att.W_value.bias, v_b)\n",
    "\n",
    "        gpt.trf_blocks[b].att.out_proj.weight = assign(\n",
    "            gpt.trf_blocks[b].att.out_proj.weight, \n",
    "            params[\"blocks\"][b][\"attn\"][\"c_proj\"][\"w\"].T)\n",
    "        gpt.trf_blocks[b].att.out_proj.bias = assign(\n",
    "            gpt.trf_blocks[b].att.out_proj.bias, \n",
    "            params[\"blocks\"][b][\"attn\"][\"c_proj\"][\"b\"])\n",
    "\n",
    "        gpt.trf_blocks[b].ff.layers[0].weight = assign(\n",
    "            gpt.trf_blocks[b].ff.layers[0].weight, \n",
    "            params[\"blocks\"][b][\"mlp\"][\"c_fc\"][\"w\"].T)\n",
    "        gpt.trf_blocks[b].ff.layers[0].bias = assign(\n",
    "            gpt.trf_blocks[b].ff.layers[0].bias, \n",
    "            params[\"blocks\"][b][\"mlp\"][\"c_fc\"][\"b\"])\n",
    "        gpt.trf_blocks[b].ff.layers[2].weight = assign(\n",
    "            gpt.trf_blocks[b].ff.layers[2].weight, \n",
    "            params[\"blocks\"][b][\"mlp\"][\"c_proj\"][\"w\"].T)\n",
    "        gpt.trf_blocks[b].ff.layers[2].bias = assign(\n",
    "            gpt.trf_blocks[b].ff.layers[2].bias, \n",
    "            params[\"blocks\"][b][\"mlp\"][\"c_proj\"][\"b\"])\n",
    "\n",
    "        gpt.trf_blocks[b].norm1.scale = assign(\n",
    "            gpt.trf_blocks[b].norm1.scale, \n",
    "            params[\"blocks\"][b][\"ln_1\"][\"g\"])\n",
    "        gpt.trf_blocks[b].norm1.shift = assign(\n",
    "            gpt.trf_blocks[b].norm1.shift, \n",
    "            params[\"blocks\"][b][\"ln_1\"][\"b\"])\n",
    "        gpt.trf_blocks[b].norm2.scale = assign(\n",
    "            gpt.trf_blocks[b].norm2.scale, \n",
    "            params[\"blocks\"][b][\"ln_2\"][\"g\"])\n",
    "        gpt.trf_blocks[b].norm2.shift = assign(\n",
    "            gpt.trf_blocks[b].norm2.shift, \n",
    "            params[\"blocks\"][b][\"ln_2\"][\"b\"])\n",
    "\n",
    "    gpt.final_norm.scale = assign(gpt.final_norm.scale, params[\"g\"])\n",
    "    gpt.final_norm.shift = assign(gpt.final_norm.shift, params[\"b\"])\n",
    "    gpt.out_head.weight = assign(gpt.out_head.weight, params[\"wte\"])\n",
    "    \n",
    "    \n",
    "load_weights_into_gpt(gpt, params)\n",
    "gpt.to(device);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "07722733",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort moves you toward finding an ideal new way to practice something!\n",
      "\n",
      "What makes us want to be on top of that?\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=gpt,\n",
    "    idx=text_to_token_ids(\"Every effort moves you\", tokenizer).to(device),\n",
    "    max_new_tokens=25,\n",
    "    context_size=NEW_CONFIG[\"context_length\"],\n",
    "    top_k=50,\n",
    "    temperature=1.5\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "build-LLM-from-scratch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
